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FOREWORD 


The  U.S.  Army  uses  advertisements  to  affect  the  knowledge,  attitudes,  and 
behavioral  intentions  of  youth  and  such  significant  influencers  as  parents  to 
effectively  recruit  manpower.  Army  advertising  development  and  execution  is 
guided  by  a  positioning  statement  and  by  specific,  measurable  objectives. 

This  final  report  documents  the  methodology  of  the  main  survey  conducted  to 
measure  the  achievement  of  those  objectives  under  the  Army  Communications 
Objectives  Measurement  System  (ACOMS) ,  which  supports  Army  assessments  of 
advertising  program  strategies  and  effectiveness.  ACOMS  also  supports  both 
planning  for  future  strategy  and  increasing  the  operational  efficiency  of  Army 
advertising  programs. 

ACOMS  was  developed  to  meet  the  needs  of  Army  policy  makers  and  opera¬ 
tional  managers  through  a  cooperative  effort  with  a  Special  Advisory  Group 
(SAG)  of  representatives  from  the  staffs  of  the  Office  of  the  Deputy  Chief  of 
Staff  for  Personnel,  the  U.S.  Army  Recruiting  Command,  the  U.S.  Army  Reserve 
Officers'  Training  Corps  (ROTC)  Cadet  Command,  the  Office  of  the  Chief  of  the 
Army  Reserve,  and  the  Army  National  Guard.  Funding  was  provided  by  the  U.S. 
Army  Recruiting  Command  with  some  funds  also  provided  by  the  U.S.  Army  ROTC 
Cadet  Command. 

The  participation  of  the  U.S.  Army  Research  Institute  (ARI)  in  this 
cooperative  effort  was  part  of  an  on-going  research  program  designed  to 
enhance  the  quality  of  Army  personnel.  This  work  is  an  essential  part  of  the 
mission  of  ARI's  Manpower  and  Personnel  Policy  Research  Group  (MPPRG)  to 
conduct  research  to  improve  the  Army's  capability  to  effectively  and  effi¬ 
ciently  recruit  its  personnel.  Specific  efforts  on  ACOMS  were  undertaken  at 
the  direction  of  the  Deputy  Chief  of  Staff  for  Personnel  (references:  Message 
2614317  NOV  84,  subject:  "Operation  Image - Watchdog, “  and  Memorandum  for 
Record,  ODCSPER,  DAPE-ZXA,  3  Feb  86,  subject:  Army  Communications  Objectives 
Survey  (ACOMS)).  The  survey  results  were  briefed  to  the  SAG  at  quarterly  in¬ 
progress  reviews  and  were  also  briefed  in  September  1987  and  April  1988  to  the 
Deputy  Chief  of  Staff  for  Personnel,  the  Commander  of  the  U.S.  Army  Recruiting 
Command,  and  the  Chief  of  the  Army  Reserve. 


EDGAR  M.  JOHNSON 
Technical  Director 
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PREFACE 


This  is  the  final  report  on  the  execution  of  the  main  data  collection 
effort  for  Project  Image  Watch-Dog,  "Army  Communications  Objectives  Measure¬ 
ment  System  (ACOMS)."  This  project  addressed  the  market  for  the  personnel 
accessioning  system  that  is  responsible  each  year  for  obtaining  from  the  non¬ 
prior-service  youth  market  over  200,000  volunteers  for  the  Army  enlisted  and 
warrant  officer  force.  In  addition,  the  U.S.  Army  Reserve  Officers'  Training 
Corps  (ROTC)  Cadet  Command  is  responsible  for  attracting  over  37,000  high- 
quality  youth  as  college  freshmen  at  4-year  colleges.  To  effectively  recruit 
in  the  youth  market,  various  components  of  the  U.S.  Army  use  advertisements  to 
produce  changes  in  the  knowledge,  attitudes,  and  behavioral  intentions  of 
youth  and  such  significant  influencers  as  peers  and  parents.  ACOMS  provides  a 
measurement  and  analysis  system  to  support  Army 

(1)  assessments  of  advertising  program  effectiveness; 

(2)  assessments  of  advertising  strategy  efficiencies; 

(3)  management  of  the  advertising  program;  and 

(4)  planning  and  development  of  new  marketing  strategies  and 
segmentation. 

The  planning  for  this  research  was  initiated  in  1984.  ACOMS  developed 
out  of  work  performed  for  a  series  of  advertising  effectiveness  conferences 
directed  by  the  U.S.  Army  Recruiting  Command  (USAREC)  at  the  request  of  the 
Deputy  Chief  of  Staff  for  Personnel.  DCSPER,  who  met  with  the  Commander  of 
USAREC,  the  Chief  of  the  Army  Reserve,  the  Director  of  the  Army  National 
Guard,  and  the  Deputy  Chief  of  Staff  of  Training  and  Doctrine  Command  for  ROTC 
in  November  1984  to  review  the  results  of  these  conferences.  These  officers 
approved  the  mission  requirements  for  ACOMS  prepared  by  their  staffs,  as  well 
as  the  basic  research  plan  prepared  by  ARI .  The  DCSPER  directed  ARI  to  de¬ 
velop  and  monitor  research  plans  and  necessary  contract  efforts  for  ACOMS  with 
guidance  from  a  Special  Advisory  Group  (SAG)  from  the  involved  Army  offices. 
The  Defense  Manpower  Data  Center  was  added  in  a  special  technical  advisory 
capacity  before  the  first  meeting  of  the  SAG. 

The  SAG  was  intimately  involved  in  refining  the  mission  requirements  for 
ACOMS  throughout  the  procurement  process  that  led  to  the  selection  in  1985  of 
Westat,  Inc.,  as  the  ACOMS  contractor.  Scientists  from  Westat  and  the  Army 
community,  together  with  a  multitude  of  advisors,  developed  and  refined  the 
research  plans  for  the  ACOMS  Survey  Design  (Nieva  &  Elig,  1988)  and  the  com¬ 
panion  volume  on  the  ACOMS  Survey  Analysis  Plan  (Gaertner  &  Elig,  1988).  In 
addition  to  guidance  from  the  SAG,  plans  for  ACOMS  benefited  from  advice  con¬ 
cerning  sampling,  weighting,  and  estimation  from  a  Statistical  Advisory  Panel. 

TIMOTHY  W.  ELIG 

ARI  Senior  Scientist  and  COR 
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THE  ARMY  COMMUNICATIONS  OBJECTIVES  MEASUREMENT  SYSTEM  (ACOMS) :  SURVEY  METHODS 


EXECUTIVE  SUMMARY 


Requirement: 

To  improve  the  efficiency  and  effectiveness  of  Army  advertising 
communications . 


Procedure : 

A  national  probability  sample  of  American  youth  and  their  parents  was 
interviewed  from  13  October  1986  through  December  1987,  using  computer- 
assisted-telephone-interviewing  (CATI)  technology.  The  Waksberg  Random  Digit 
Dialing  (RDD)  method  was  used  to  locate  eligible  youths,  defined  as  noninsti- 
tutionalized  16-  to  24-year-old  youth,  living  in  the  contiguous  48  states,  who 
have  no  prior  military  service  or  contractual  commitment  to  serve  in  the  mili¬ 
tary  and  who  are  not  graduates  of  4-year  colleges.  Attempts  were  made  to 
interview  all  eligible  male  youth  located  and  a  subsample  of  female  youth. 

In  response  to  concerns  for  fostering  equal  opportunity  in  the  military, 
the  ACOMS  sample  was  designed  to  obtain  a  sample  that  would  allow  analysis  of 
the  major  racial/ethnic  groups,  i.e.,  blacks  and  Hispanics .  A  supplementary 
sample  of  Hispanic  males  were  interviewed  to  meet  these  goals.  Since  a  suf¬ 
ficiently  large  sample  of  blacks  was  expected  from  the  main  sample,  no  black 
supplementation  was  conducted.  In  addition  to  the  youth  sample,  which  was  the 
primary  focus  of  the  survey,  interviews  were  conducted  with  a  sample  of  par¬ 
ents  of  primary  sample  youth  between  16  to  20  years  of  age. 

Three  survey  instruments  were  used:  a  household  screening  interview 
designed  to  locate  youth  eligible  for  participation  in  the  ACOMS  survey;  a 
youth  interview;  and  a  parental  interview.  All  youth  respondents  were  ques¬ 
tioned  regarding  a  variety  of  issues  related  to  advertising  and  the  enlistment 
decision  process,  e.g.,  their  media  habits,  knowledge  about  various  Army 
components,  perceptions  of  Army  attributes,  and  enlistment  intentions  and 
behaviors . 

Based  on  listings  of  telephone  numbers,  monthly  samples  of  youth  and 
their  parents  were  drawn.  The  data  collection  period  for  each  monthly  youth 
sample  was  8  weeks,  and  data  for  their  parents  were  collected  up  to  4  weeks 
after  the  youth  interview.  A  large  staff  of  trained  interviewers  conducted 
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interviews  7  days  a  week.  Modifications  were  made  incrementally  to  the  data 
collection  procedures  to  achieve  desired  sample  sizes.  Data  collection  pro¬ 
cedures  had  stabilized  by  the  fall  1987. 


Findings : 

Screening  interviews  of  106,419  households  were  conducted  from  October 
1986  through  December  1987;  13,197  youth  interviews  and  4,144  parental  inter¬ 
views  were  also  conducted. 

Household  screenings  were  completed  in  83X  of  sampled  households.  Inter¬ 
views  were  completed  with  76%  of  eligible  youth  identified  and  61.5%  of  sam¬ 
pled  parents . 

All  data  were  weighted  up  to  the  eligible  U.S.  population  on  a  quarterly 
basis . 


Utilization  of  Findings: 

ACOMS  data  provide  information  useful  to  Army  policy  makers  and  organiza¬ 
tions  with  operational  responsibility  to  more  effectively  recruit  in  the  youth 
market.  ACOMS  data  provide  the  Army  with  the  opportunity  to  examine  questions 
and  to  develop  policies  key  to  the  recruiting  effort  by  using  a  nationally 
representative  data  base  that  was  developed  from  state-of-the-art  scientific 
methodology.  ACOMS  is  being  used  for  Army 

(1)  assessments  of  advertising  program  effectiveness; 

(2)  assessments  of  advertising  strategy  efficiencies; 

(3)  management  of  the  advertising  program;  and 

(4)  planning  and  development  of  new  marketing  strategies  and 
segmentations . 

Based  on  ACOMS  analyses,  the  Reserve  Officers'  Training  Corps  Cadet 
Command  has  changed  the  attributes  to  be  emphasized  in  future  marketing 
efforts  for  Army  officers.  In  addition,  the  relative  emphasis  placed  on  Army 
attributes  is  being  changed  in  advertising  efforts  for  the  enlisted  force. 
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THE  ARMY  COMMUNICATIONS  OBJECTIVES  MEASUREMENT  SYSTEM 
(ACOMS)  SURVEY  METHODOLOGY 

1.  INTRODUCTION 

Veronica  F.  Nieva  and  Timothy  W.  Elig 

The  Army  Communications  Objectives  Measurement  System  (ACOMS) 
survey  was  designed  to  provide  timely  information  to  Army  policy¬ 
makers  and  advertising  planners  regarding  key  market  responses  to  the 
Army's  advertising  programs.  Survey  data  were  collected  continuously 
from  a  national  probability  sample  of  American  youth  and  their  par¬ 
ents  during  the  period  starting  October  1986  until  December  1987. 
Youth  respondents  were  located  using  the  Waksberg  Random  Digit  Dial¬ 
ing  (RDD)  method.  Interviews  were  conducted  using  computer  assisted 
telephone  interviewing  (CATI)  technology. 

This  report  documents  the  methodology  used  in  conducting  the 
survey.  Modifications  from  the  original  survey  design  that  were 
implemented  throughout  the  course  of  the  survey  and  the  rationale  and 
results  of  such  changes  are  also  discussed.  This  report  is  one  of  a 
series  that  describes  various  aspects  of  the  ACOMS  project:  the 
ACOMS  Survey  Design  (Nieva  and  Elig,  1988);  the  ACOMS  Analysis  Plan 
(Gaertner  and  Elig,  1988);  substantive  analysis  based  on  the  data 
collected  for  the  school  year  1986/87  (Nieva,  Gaertner,  Elig  and 
Benedict  1988);  a  quarterly  report  series  (Gaertner,  Nieva,  Elig  and 
Benedict,  1988);  and  two  volumes  of  supplementary  tabulations  for  the 
enlisted  and  officer  markets  (Rhoads,  Elig,  McEntire,  and  Hoke, 

1988). 

The  remainder  of  this  introductory  chapter  provides  an  overview 
of  ACOMS  objectives  and  its  major  design  elements.  Chapter  2 
describes  the  sample  yields  obtained  in  the  ACOMS  survey.  Chapter  3 
describes  the  content  and  structure  of  the  youth  and  parental  ques¬ 
tionnaires.  General  descriptions  of  the  types  of  questionnaire 
changes  made  over  the  course  of  the  survey  are  included;  however,  for 
a  detailed  inventory  of  specific  changes,  the  reader  is  referred  to 
the  ACOMS  Users'  Manuals  (Westat,  1988).  Chapter  4  discusses  the 
preparation  for  and  actual  conduct  of  the  data  collection  effort. 
Chapter  5  describes  the  weighting  methodology,  survey  design  effects, 
and  estimation  procedures  used  in  the  Acorns  analyses. 


^COMS  Objectives 


In  an  era  when  manpower  experts  are  predicting  increased  diffi¬ 
culties  for  recruiting  into  military  service,  it  becomes  increasingly 
important  for  the  U.S.  Army  to  improve  understanding  and  management 
of  the  factors  that  enable  it  to  meet  its  manpower  goals.  Advertis¬ 
ing  communications  represent  one  such  factor. 


Advertising  is  used  extensively  by  the  various  Army  components- - 
the  Active  Army,  the  U.S.  Army  Reserve  (USAR) ,  the  Army  Reserve 
Officers'  Training  Corps  (ROTC) ,  and  the  Army  National  Guard  (ARNG)-- 
to  induce  changes  in  the  knowledge,  attitudes,  and  behaviors  of  youth 


1 


and  their  parents.  Each  year,  the  Army  makes  a  sizable  investment  in 
the  development  and  exposure  of  advertising  communications  intended 
to  disseminate  favorable  Army  images  and  to  increase  the  enlistment 
propensity  of  eligible  youth. 

While  in-house  assessments  have  been  conducted  by  the  Army's 
main  advertising  contractor,  and  advertising  has  been  touched  upon  by 
other  youth  surveys  such  as  the  Youth  Attitude  Tracking  Survey 
(YATS)  ,  there  had  been  no  in-depth,  independent  examination  of  the 
effectiveness  of  Army  communications  prior  to  the  ACOMS  effort. 

ACOMS  was  designed  to  help  the  Army  monitor  and  evaluate  its  adver¬ 
tising  communications  program  and  to  provide  ongoing  measurement  of 
the  extent  to  which  Army  communications  meet  the  communications 
objectives  for  different  target  groups. 

The  objectives  of  ACOMS  were: 

(1)  To  support  Army  assessments  of  advertising  program 
effectiveness  in  a  timely  fashion; 

(2)  To  support  Army  assessments  of  advertising  strategy  in  an 
integrated  framework;  and 

(3)  To  support  Army  advertising  management  and  planning  for 
future  strategy. 

ACOMS  was  designed  to  monitor  and  assess  the  effectiveness  of 
the  Army's  advertising  communications  program.  A  continuous  survey 
data  collection  effort  was  designed  to  track  changes  over  time  in 
levels  of  advertising  recall,  and  subsequent  effects  on  the  knowl¬ 
edge,  attitudes,  intentions,  and  actions  of  youth  and  their  parents. 
Periodic  analysis  of  key  market  responses  could  be  reported  to  the 
Army  decisionmakers  and  advertising  planners  in  a  timely  fashion. 

The  second  and  third  goals  for  ACOMS  involved  the  use  of  ACOMS 
data  to  assess  the  Army's  advertising  strategy  and  to  support  manage¬ 
ment  and  planning  for  future  strategy.  ACOMS  survey  data  included  an 
integrated  set  of  indicators  that  were  intended  to  be  influenced  by 
Army  advertising,  i.e.,  youth  and  parental  recall  of  Army  advertise¬ 
ments,  knowledge  of  Army  offers,  perceptions  about  the  Army,  and 
enlistment  related  intentions  and  behaviors.  At  any  one  point  in 
time,  assessments  of  Army  advertising  could  be  conducted  by  analysis 
of  these  indicators.  In  addition,  a  longitudinal  survey  was  designed 
to  be  part  of  ACOMS  to  further  the  understanding  of  the  lagged  and 
cumulative  effects  of  advertising  and  of  the  relationships  between 
advertising  and  other  factors  that  influence  the  eventual  enlistment 
decision. 

Thus,  ACOMS  was  intended  to  contribute  to  the  development  of 
behavioral  and  economic  models  of  enlistment  decisionmaking  currently 
being  developed  by  the  Army  Research  Institute.  As  better  models  of 
the  enlistment  decision  process  are  developed,  more  effective  market¬ 
ing  strategies  can  be  applied  to  help  the  Army  attain  its  annual 
recruiting  goals. 
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The  targeting  of  advertising  to  the  Army's  recruiting  market 
segments  was  another  aspect  of  advertising  strategy  that  is  supported 
by  ACOMS  survey  data.  Analysis  of  reactions  to  advertising,  media 
habits,  attitudes,  and  other  variables  by  major  demographic  segments 
of  interest  and  regions  can  refine  the  definitions  of  the  Army's 
major  market  segments  or  identify  and  validate  new  market  segments. 

ACOMS  survey  data  were  also  intended  to  support  development  of 
the  Army's  advertising  strategy  by  allowing  analyses  on  brand  differ¬ 
entiation  ("brands"  referring  to  various  Army  components  or  the 
different  military  services).  Youth  perceptions  of  various  Army  or 
military  "brands"  would  be  useful  in  examining  brand  differentiation 
at  several  levels:  differentiation  among  the  active  Army,  Reserve, 
National  Guard,  and  ROTC  attributes;  differentiation  between  the  Army 
and  other  services'  attributes;  and  differentiation  between  the 
Army's  position  and  distinctive  advantages  vis-a-vis  civilian  alter¬ 
natives  (i.e.,  college  and  civilian  employment).  This  information 
helps  the  Army  make  decisions  on  relative  emphasis  of  various 
communications  about  different  attributes  and  offers  of  the  Army 
components . 

Finally,  development  of  advertising  strategy  was  supported  by  a 
separate  ACOMS  data  collection  effort.  Mall  intercept  interviews 
were  collected  in  order  to  examine  the  extent  to  which  the  Army's 
intended  messages  are  actually  exposed  to,  and  perceived  by,  their 
target  audiences.  Description  of  this  element  of  ACOMS  is  presented 
in  Baxter  and  Gay  (1988) . 

The  ACOMS  Survey  Overview 

The  continuous  survey  of  American  youth  and  their  parents 
constituted  the  core  of  ACOMS.  As  noted  above,  the  survey  was 
designed  to  support  all  of  the  ACOMS  project  objectives.  This 
section  presents  an  overview  of  the  major  survey  elements.  Further 
detail  on  how  the  survey  was  actually  conducted  is  presented  in 
Chapters  2  to  5  of  this  report. 

The  ACOMS  Sample 

National  probability  samples  of  American  youth  and  their  parents 
were  interviewed  from  13  October  1986  through  December  1987,  using 
computer  assisted  telephone  interviewing  (CATI)  technology.  The 
Waksberg  Random  Digit  Dialing  (RDD)  method  was  used  to  locate 
eligible  youths,  defined  as  non- institutionalized  16-  to  24-year-old 
youth,  living  in  the  contiguous  48  states  with  no  prior  military 
service  nor  contractual  commitment  to  serve  in  the  military,  and  who 
are  not  graduates  of  four-year  colleges. 

Attempts  were  made  to  interview  all  eligible  male  youth  located 
and  a  subsample  of  female  youth.  Thus,  the  first  important  distinc¬ 
tion  made  in  the  youth  sample  was  gender.  The  sample  was  also 
designed  to  be  sensitive  to  educational  categorizations,  because  of 
t^e  Army's  interest  in  recruiting  Tier  1  youth,  i.e.,  youth  with 
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regular  high  school  diplomas  or  youth  with  at  least  one  semester  of 
college  credit.  The  ACOMS  Primary  sample  (PS)  consisted  of  male 
(Primary  Male  Sample,  or  PMS)  and  female  (Primary  Female  Sample,  or 
PFS)  youth  in  the  Tier  1  priority  group.  The  ACOMS  Secondary  sample 
(SS)  consisted  of  youth  who  were  holders  of  Adult  Basic  Education 
(ABE)  or  General  Educational  Development  (GED)  certificates,  or  youth 
who  did  not  complete  high  school,  did  not  have  an  ABE  or  GED,  and 
were  not  enrolled  in  regular  high  school  or  college.  These  youth  are 
also  referred  to  as  Tier  2  and  Tier  3  youth  in  the  military's  system 
of  educational  priorities.  Males  in  these  categories  are  referred  to 
as  the  Secondary  Male  sample  (SMS)  and  females  as  the  Secondary 
Female  sample  (SFS). 

In  response  to  concerns  for  fostering  equal  opportunity  in  the 
military,  the  ACOMS  sample  was  designed  to  obtain  a  sample  that  would 
allow  analysis  of  the  major  racial/ethnic  groups,  i.e.,  Blacks  and 
Hispanics.  A  supplementary  sample  of  Hispanic  males  was  interviewed 
in  order  to  meet  these  goals.  Since  a  sufficiently  large  sample  of 
Blacks  was  expected  from  the  main  sample,  no  Black  supplementation 
was  conducted. 

In  addition  to  the  youth  sample ,  which  was  the  primary  focus  of 
the  survey,  interviews  were  conducted  with  a  sample  of  parents  of 
primary  sample  youth  between  16  to  20  years  of  age.  Parents  were  of 
interest  in  large  part  because  of  their  potential  influence  on  the 
military  enlistment  decisions  of  their  sons  and  daughters.  They  were 
also  perceived  as  a  potential  direct  target  for  Army  advertising. 

Initial  sample  yields  were  lower  than  expected.  Thus,  modifica¬ 
tions  were  made  incrementally  to  the  survey  procedures  in  order  to 
achieve  desired  sample  sizes.  Sample  yield  goals  had  been  reached  by 
the  fall  of  1986.  Details  of  the  sample  and  modifications  made  to 
improve  yield  are  presented  in  Chapter  2. 

The  ACOMS  Survey  Instruments 

Three  survey  instruments  were  used:  a  household  screening 
interview  designed  to  locate  youth  eligible  for  participation  in  the 
ACOMS  survey;  a  youth  interview;  and  a  parental  interview. 

All  youth  respondents  were  questioned  regarding  a  variety  of 
issues  related  to  advertising  and  the  enlistment  decision  process, 
e.g.,  their  media  habits,  knowledge  about  various  Army  components, 
perceptions  of  Army  attributes  and  enlistment  intentions  and  behav¬ 
iors.  The  large  numbers  of  content  domains  included  in  the  survey 
instrument  had  to  be  balanced  by  pragmatic  concerns  respondent 
burden,  necessitating  a  complex  instrument  with  extensive  branching 
and  allocation  of  questions  to  subpopulations  of  respondents. 

The  parental  survey  instrument  largely  paralleled  the  youth 
instrument  in  both  form  and  content.  Details  on  the  survey  instru¬ 
ments  are  included  in  Chapter  3. 


4 


The  ACOMS  Survey  Data  Collection 


Survey  data  were  collected  from  13  October  1986  through  December 
1987.  CATI  technology  was  used  to  interview  monthly  samples  of  youth 
and  their  parents.  The  Waksberg  RDD  method  was  used  to  locate  house¬ 
holds  with  eligible  youth.  In  addition  to  the  youth  and  parents  in 
the  main  sample,  a  supplementary  Hispanic  male  sample  was  drawn, 
applying  the  RDD  method  to  a  sample  of  telephone  exchanges  known  to 
have  high  Hispanic  density. 

The  data  collection  period  for  each  monthly  youth  sample  was 
eight  weeks,  and  data  for  their  parents  were  collected  up  to  four 
weeks  after  the  youth  interview.  A  large  staff  of  trained  inter¬ 
viewers  conducted  interviews  seven  days  a  week. 

Chapter  4  describes  the  survey  data  collection  effort. 

The  ACOMS  Sample  Weighting.  Design  Effects  and  Analysis  Methodology 

In  order  to  achieve  desired  sample  sizes  of  respondents  in 
different  subgroups ,  ACOMS  youth  and  parental  respondents  were 
selected  with  differing  probabilities  of  selection.  This  deviation 
from  simple  random  sampling  (SRS)  procedures  necessitated  the  devel¬ 
opment  of  sample  weights  in  order  to  produce  unbiased  estimates  of 
ACOMS  population  parameters.  Further,  the  departure  from  an  SRS 
design,  while  improving  the  effectiveness  of  data  collection,  had 
some  costs  in  terms  of  sample  efficiency.  This  relative  efficiency 
is  called  the  sample  "design  effect."  A  final  implication  of  the 
complex  sample  design  is  the  need  to  use  appropriate  tests  of  statis¬ 
tical  significance.  ACOMS  analyses  used  two  primary  approaches  for 
testing  statistical  significance:  approximate  chi-square  statistics 
and  the  balanced  repeated  replication  (BRR)  methodology. 

Chapter  5  describes  the  sample  design  effects  and  the  method¬ 
ology  used  for  sample  weighting  and  estimation. 
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2.  ACOMS  SURVEY  SAMPLE 


Leyla  Mohadjer  and  Veronica  F.  Nieva 

The  Army  Communications  Objectives  Measurement  Survey  (ACOMS) 
required  the  design  and  implementation  of  a  major  survey  aimed  at 
evaluating  the  effectiveness  of  Army  advertising  communications  over 
time.  Several  changes  were  made  to  the  original  basic  and  optional 
samples,  given  in  the  Statement  of  Work  (SOW),  as  a  result  of  a 
series  of  meetings  with  the  ACOMS  Special  Advisory  Group  (SAG)  and 
the  statistical  external  panel.  Most  changes  involved  sample  reduc¬ 
tions  in  response  to  SAG  priorities  and  budgetary  concerns.  The 
decisions  to  make  these  changes  were  reached  on  the  basis  of  a  series 
of  studies  that  investigated  the  cost  and  benefit  of  each  of  the 
basic  and  optional  samples.  Refer  to  the  ACOMS  Survey  Design  (Nieva 
&  Elig,  1988)  for  information  about  the  original  design  requirements 
and  the  studies.  The  ACOMS  sample  design  incorporated  these  revised 
requirements . 


Sample  Design  Overview 


Respondent  Universe 

Youth  sample.  The  main  respondent  universe  for  this  survey 
consisted  of  eligible  youth  in  households  with  telephones  in  the  48 
contiguous  states.  Eligibility  was  defined  in  terms  of  age  and 
service  in  the  military,  as  described  below.  The  universe  of  eligi¬ 
ble  youth  was  divided  further  by  gender  and  educational  categories . 
The  major  youth  sample  groups  were: 

(1)  Primary  Male  Sample  (PMS):  16-  to  24-year-old  males  who 
were  in  the  Army's  Tier  1  priority  group,  i.e.,  those  who 
were  in  high  school  or  who  were  high  school  diploma  (HSD) 
graduates  but  not  graduates  of  four-year  colleges, 
excluding  in-service  and  prior-service  population.  The  PMS 
definition  also  excluded  youth  with  General  Education 
Diploma/Adult  Basic  Education  (GED/ABE) ,  or  any  other 
"nonregular"  certificate  of  high  school  completion. 

(2)  Secondary  Male  Sample  (SMS):  16-  to  24-year-old  males  who 
were  in  the  Army's  Tiers  2  and  3,  i.e.,  those  who  were  not 
in  high  school  and  who  did  not  have  an  HSD,  excluding  in- 
service  and  prior- service  population.  The  SMS  sample 
included  males  with  GED/ABE  or  other  "non- regular" 
certificate  of  high  school  completion. 

(3)  Female  Sample  (FS),  comprising  Primary  (PFS)  and  Secondary 
(SFS)  samples,  with  similar  definitions  as  given  for  the 
male  samples. 

Parental  sample.  The  respondent  universe  also  included  parents 
of  the  sampled  PMS  and  PFS  who  were  16-  to  20-years  of  age. 
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The  Waksberg  Random  Digit  Dial  (RDD)  procedure.  The  sample  was 
selected  using  the  "modified"  Waksberg  Random  Digit  Dialing  (RDD) 
procedure.  This  method  provides  an  unbiased  representative  sample  of 
eligible  persons  in  households  with  telephones  in  a  way  that  mini¬ 
mizes  the  number  of  out-of- scope  telephone  numbers  that  had  to  be 
screened.  Essentially,  the  modified  Waksberg  RDD  procedure  involves 
two  steps:  first,  "household  cluster  identification"  (identifying 
and  selecting  a  sample  of  blocks  of  100  numbers  called  "telephone 
clusters,"  which  contain  working,  residential  telephone  numbers);  and 
second,  dialing  random  numbers  within  the  clusters  to  locate  house¬ 
holds  of  eligible  respondents. 

The  ACOMS  sampling  frame  consisted  of  the  most  recent  AT&T  tape 
containing  all  existing  telephone  area  codes  and  existing  exchanges. 
The  frame  was  updated  every  six  months.  Details  of  the  procedures 
for  RDD  sample  selection  are  described  in  Waksberg  (1978)  and  in  the 
ACOMS  Survey  Design  report  (Nieva  &  Elig,  1988). 

For  each  telephone  number  contacted  within  the  household 
clusters ,  a  brief  screening  interview  was  conducted  in  order  to 
determine  (a)  whether  the  number  was  located  in  a  business  or  house¬ 
hold  and,  if  in  a  household,  (b)  whether  youth  meeting  the  necessary 
criteria  for  inclusion  in  the  study  were  members  of  the  household. 

Any  member  of  the  household  over  the  age  of  16  was  eligible  to 
respond  to  the  screener  questions. 

If  youth  meeting  the  criteria  for  selection  were  identified, 
attempts  were  made  to  complete  those  interviews  within  the  eight  week 
field  period  for  that  set  of  telephone  numbers.  All  male  youth  and 
about  one -fifth  of  the  female  youth  who  met  the  ACOMS  eligibility 
criteria  were  selected  for  the  full  interview.  An  additional  four 
weeks  from  the  time  of  the  completed  youth  interview  was  allowed  for 
the  completion  of  an  interview  with  the  parental  sample. 

Sample  size  goals.  Annual  sample  size  targets  for  various 
subgroups  of  interest  were  established  in  the  SOW.  As  noted  previ¬ 
ously,  the  main  group  of  interest  was  the  Army's  Tier  1  male  group, 
otherwise  referred  to  as  the  Primary  Male  Sample  (PMS).  No  specific 
targets  were  set  in  the  SOW  for  the  Tier  2  and  3  males,  otherwise 
referred  to  as  the  Secondary  Male  Sample  (SMS).  This  group  was 
considered  a  "convenience"  sample,  and  the  targets  were  defined  as 
the  number  of  such  youth  who  were  located  in  the  households  with  PMS 
youth . 


The  SOW  also  set  target  sample  sizes  for  other  groups  of 
interest:  females,  youth,  black  males ,  Hispanic  males,  and  PMS 

enrolled  in  two  and  four  year  colleges.  Table  1  shows  the  original 
target  sample  sizes  as  well  as  revised  targets  established  after 
changing  the  definition  of  the  PMS  and  SMS  groups.  The  PMS  was 
originally  defined  as  those  youth  who  were  in  school  or  high  school 
graduates  but  not  4  year  college  graduates.  After  the  start  of  the 
study,  it  was  decided  that  the  PMS  should  exclude  not- in-school  youth 


Table  1 


Original  and  Revised  Annual  Target  Number  of  Respondents 


Subpopulation 

Old  Target 

Revised 

New  Target3 

Primary  Male  Sample  (PMS) 

9,603 

9,300 

Female  Sample  (FS) 

1,953 

1,953 

Black 

1,200 

1,162 

Hispanic 

970 

970 

Secondary  Male  Sample  (SMS) 

850 

1,153 

Total  males 

10,753 

10,753 

Total  youth 

12,706 

12,706 

aNew  targets  established  Summer  of  1986  in  clarification  that 
General  (GED)  and  (ABE)  students  are  SMS,  not  PMS 
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with  non- traditional  certificates  of  high  school  completion,  such  as 
GED  or  ABE.  Excluding  GED/ABE  from  the  definition  of  PMS  reduced  the 
size  of  the  PMS  by  approximately  3 2  (i.e.,  approximately,  32  of  the 
total  eligible  youth  had  a  GED/ABE,  or  some  other  "non-regular" 
certificate  of  high  school  completion). 

Table  2  shows  the  estimated  number  of  telephone  clusters, 
numbers,  and  households  necessary  to  yield  the  required  sample  sizes 
for  the  various  groups  of  interest.  Estimates  were  based  on  popula¬ 
tion  data  (e.g.,  1980  census  information  on  population  distribution, 
school  enrollment  and  educational  attainment,  extrapolated  to  1986, 
data  on  the  percent  of  youth  with  prior  service,  by  age  and  race), 
estimates  of  response  rates,  and  assumptions  on  how  these  factors 
interact,  e.g.,  that  the  response  rate  would  be  the  same  in  all 
Brigade  regions  and  for  all  ages,  and  that  there  would  be  no  differ¬ 
ential  telephone  coverage  problems  for  sample  subgroups.  Response 
rate  estimates  were  based  on  a  combined  screener  and  main  interview 
response  rate  of  about  802,  as  stipulated  by  the  statement  of  work. 
The  assumed  combined  response  rate  was  broken  down  into  its  two 
components:  952  for  the  household  screener  and  852  for  the  main 
interview.  A  more  detailed  discussion  of  these  assumptions  is 
provided  in  the  ACOMS  Survey  Design  (Nieva  &  Elig,  1988). 

Households  screened  for  PMS  youth  were  expected  to  yield  the 
required  sample  sizes  for  these  other  population  groups  of  interest, 
with  the  exception  of  Hispanic  male  youth.  In  order  to  obtain  the 
required  sample  numbers  of  this  relatively  rare  group,  a  supplemental 
sample  was  drawn  from  telephone  exchanges  known  to  have  high  Hispanic 
density. 

Hispanic  supplement.  The  Hispanic  supplement  was  drawn  using  a 
data  tape  of  telephone  exchanges  and  their  associated  demographic 
characteristics  on  the  1980  Census,  developed  by  the  Donnelley 
Marketing  Information  Services.  The  tape  was  used  to  identify 
exchanges  with  heavy  concentrations  of  Hispanics.  Telephone 
exchanges  in  the  First,  Fifth,  and  Sixth  Recruiting  brigades  with 
heavy  concentration  of  Hispanics  were  identified  and  oversampled. 
Within  these  exchanges,  the  same  RDD  procedures  used  for  the  main 
sample  were  applied. 

Parental  selection.  One  parent  in  each  household  with  PMS  or 
PFS  youth  between  the  ages  of  16  and  20  was  selected  for  interview. 
For  a  randomly  chosen  half  of  the  households  with  a  16-  to-20-year- 
old  PMS/PFS  youth,  the  eligible  parent  was  female;  for  the  other 
half,  the  parent  was  male.  Only  one  parental  interview  was  conducted 
within  a  single  household.  Parental  respondents  were  selected  for 
the  parental  interview  in  a  three-stage  process.  First,  prior  to  any 
household  contact,  the  CATI  system  randomly  designated  the  gender  of 
any  potential  parent  respondent.  The  second  stage  occurred  following 
completion  of  the  screener  in  households  with  at  least  one  eligible 
youth.  At  this  point,  one  youth  was  selected  randomly  from  all  16- 
to  20-year-old  PMS  or  PFS  youth  in  the  household  to  be  the  "target” 
of  the  parental  interview.  The  youth  selected  at  this  point  were 
designated  as  the  "parental -linked  sample." 
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Table  2 


Annual  Estimated  Sample  Sizes  for  Telephone  Clusters,  Numbers, 
and  Households 


Main 

Sample 

N 

Hispanic 

Supplement 

N 

Total 

N 

Telephone  Clusters 

2,400 

192 

2,592 

Telephone  Numbers 

120,000 

11,064 

131,064 

Identified  Households 

70,000 

6,638 

76,638 

Screened  Households 

66,500 

6,306 

72,806 

In  the  third  stage,  the  appropriate  parental  respondent  was 
identified  by  the  parental -linked  youth.  In  the  demographic  and 
parental  location  modules  of  the  youth  interview,  potential  parental 
influencer(s)  of  the  predesignated  gender  were  enumerated.  Youth  who 
lived  with  a  stepparent  or  guardian  were  prompted  to  choose  from 
among  their  (a)  natural  or  adoptive  parent,  (b)  stepparent,  or 
(c)  guardian,  depending  on  their  perceptions  of  their  most  influen¬ 
tial  parental  figure.  The  parental  influencer  was  not  required  to 
live  in  the  same  household  to  be  interviewed. 

Youth  Sample  Yields 

Study  objectives  required  careful  control  of  the  sample  sizes 
for  a  variety  of  population  subgroups.  Careful  monitoring  of  sample 
yields  was  necessary  to  detect,  early  on,  whether  assumptions  under¬ 
lying  the  sample  design  would  hold.  Care  was  necessary  since  some 
assumptions  were  based  on  data  from  the  1980  Census  Bureau  that  were 
seven  years  old,  and  some  data  required  to  refine  the  sampling  plan 
(e.g.,  prior  service  estimates  and  coverage  differences  for  youth: 
with  different  education  and  prior- service  status)  were  not  available 
for  planning  purposes. 

Tables  3  to  6  show  the  major  data  collection  statistics  moni¬ 
tored  regularly  for  the  youth  sample,  i.e.,  sample  sizes  (telephone 
clusters,  numbers,  household  screener  interviews,  and  eligible  youth 
identified  and  interviewed)  and  ratios  obtained  between  key  elements 
of  the  sampling  plan  (residency  rates,  response  rates  for  households 
and  youth,  and  household  screener  to  youth  respondent  eligibility 
ratios) .  Table  3  shows  the  planned  and  actual  quarterly  sample 
sizes,  while  Table  4  shows  planned  and  actual  ratios  relevant  to 
sample  performance.  Tables  5  and  6  parallel  Tables  3  and  4,  but  show 
monthly  figures  rather  than  quarterly. 

i * 

It  should  be  noted  that  all  of  these  statistics  are  reported 
according  to  sample  cohorts  rather  than  date  of  interview.  After 
monthly  samples  of  telephone  numbers  were  drawn,  eligible  youth 
identified  were  available  for  interviews  for  a  period  of  eight 
weeks.  Thus,  sample  statistics  based  on  sample  cohorts,  such  as 
those  reported  in  Tables  3  to  6 ,  do  not  resemble  statistics  based  on 
date  of  interview,  such  as  those  reported  in  the  ACOMS  Quarterly 
Reports  (Gaertner,  Nieva,  Elig,  &  Benedict,  1988). 

The  first  quarter  of  data  collection  (Fall  1986)  shown  in 
Table  3  fell  considerably  short  of  expectations.  The  shortfall  was 
detectable  at  all  stages  of  the  survey  process --the  number  of  house¬ 
hold  screeners  completed  (17,258)  was  95X  of  the  original  plan;  and 
overall,  the  number  of  youth  interviews  completed  (2083)  was  only  66X 
of  plan. 
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Data  Collection  Ratios  by  Quarterly  Sanple  Cohort 
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Hispanic  sale  eligibility  ratio  is  derived  fro*  the  conblnstlon  of  the  basic  saaple  and  the  Hispanic 
suppleaent  f  roa  areas  with  high  Hispanic  density.  Therefore,  this  ratio  does  not  represent  expectations 
froa  the  nationwide  population. 


Monthly  Data  Collection  Totals 
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Monthly  Data  Collection  Ratios 


The  shortfall  in  the  number  of  households  screened  was  critical 
because  it  ultimately  restricted  the  size  of  the  pool  from  which 
youths  could  be  identified  and  interviewed.  The  difference  from  plan 
was  due  to  the  difference  between  the  obtained  household  response 
rate  (84. IX)  and  the  household  response  rate  assumed  in  the  original 
plan  (95X) .  In  retrospect,  the  assumption  of  a  95X  household 
response  rate  was  an  overoptimistic  base  for  realistic  planning.  The 
project  statement  of  work  required  an  overall  80X  response  rate. 

Plans  to  achieve  this  overall  response  rate  further  assumed  that  the 
household  screener  response  rate  would  be  95X,  and  that  the  youth 
response  rate  would  be  85X. 

In  addition  to  the  household  shortfalls  due  to  response  rates , 
youth  sample  yields  suffered  from  two  other  major  factors:  fewer 
youth  met  the  eligibility  criteria  than  anticipated,  and  response 
rates  for  the  extended  youth  interview  were  lower  than  planned. 

The  eligibility  ratio  for  each  major  youth  group  was  defined  in 
terms  of  the  ratio  between  the  number  of  households  successfully 
screened  and  the  number  of  youth  identified  who  met  the  criteria  for 
inclusion.  The  sampling  plan  expected  an  overall  youth  eligibility 
ratio  of  4.9  households  to  1  youth  identified,  and  a  PMS  eligibility 
ratio  of  5.9  households  to  1  PMS  identified.  In  the  first  quarter  of 
data  collection  the  overall  youth  eligibility  ratio  obtained  was  6.1 
households  to  1;  for  PMS,  it  was  8.6  households  to  1.  The  number  of 
youth  eligibles  located  in  the  first  quarter  (N-2,829)  was  only  76X 
of  plan,  with  a  somewhat  larger  shortfall  for  the  Primary  Male  Sample 
(71X  of  plan).  Planned  and  actual  eligibility  rates  for  various 
youth  subgroups  are  shown  in  Table  4. 

A  number  of  factors  could  potentially  explain  shortfalls  in 
eligibility  ratios,  although  no  definitive  information  was  available 
to  reach  firm  conclusions.  They  include:  parental  gatekeeping 
(i.e.,  parents  who  did  not  enumerate  eligible  youth  in  order  to  avoid 
further  participation  in  the  survey);  lower  screener  response  rates 
among  households  with  PMS  youth  than  other  households;  and  telephone 
undercoverage,  particularly  among  older  youths  (i.e.,  21-  to  24-year- 
old  youth) . 

The  only  group  for  which  there  was  no  serious  eligibility  prob¬ 
lems  was  the  Secondary  Male  Sample,  which  had  a  much  higher  eligibil¬ 
ity  rate  than  expected.  In  part,  this  could  be  attributed  to  the 
changes  made  in  eligibility  criteria  in  the  Spring  1987  quarter 
which  moved  ABE  and  GED  high  school  completions  from  the  PMS  to  the 
SMS  group. 

Sample  yield  was  affected,  not  only  by  the  household  response 
rate  and  the  youth  eligibility  ratio,  but  also  by  the  youth  response 

*"The  household  screener  response  rate  was  defined  as  the  number  of 
household  screeners  completed  divided  by  the  number  of  households 
identified.  The  youth  response  rate  was  defined  as  the  number  of 
youth  interviewed  divided  by  the  number  of  youth  identified. 
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races.  The  youth  response  rate  obtained  in  the  first  quarter  was 
74X,  lower  than  the  expected  level  of  85Z.  While  this  response  rate, 
like  that  obtained  for  household  interviews,  was  good  and  comparable 
to  those  obtained  by  similar  studies  such  as  the  Youth  Attitude 
Tracking  Survey,  Phase  III  (78. 8X),  it  was  lower  than  the  optimistic 
rate  stipulated  by  the  original  statement  of  work,  which  was  used  as 
the  basis  for  designing  the  sampling  and  data  collection  plan. 

Modifications  to  Improve  Youth  Sample  Yields 

A  series  of  incremental  measures  were  therefore  undertaken  to 
improve  both  eligibility  and  response  rates.  Four  types  of  modifica¬ 
tions  were  undertaken:  (a)  changes  in  household  screener  wording; 

(b)  optimizing  interviewer  staffing  patterns  by  day  of  the  week  and 
time  of  day;  (c)  changes  in  eligibility  determination  procedures;  and 
(d)  increasing  the  sample  of  telephone  numbers  contacted  per  month. 
The  first  two  modifications  were  conducted  just  prior  to  the  start  of 
the  second  quarter  of  data  collection.  The  last  two  modifications 
were  started  at  the  beginning  of  the  third  quarter,  although  the 
increase  in  telephone  numbers  sampled  continued  incrementally  until 
the  fall  of  1987.  Each  of  these  modifications  and  their  effects  are 
discussed  below. 

Household  screener  changes.  Since  the  majority  of  the  refusals 
to  the  household  screener  occurred  at  the  start  of  the  interview,  the 
introduction  was  seen  as  a  critical  factor  for  keeping  the  respondent 
on  the  telephone.  The  following  wording  change  was  made  to  allay 
respondent  apprehension  about  the  sponsorship  and  content  of  the 
interview. 


Original  Version 

Hello,  this  is  (Interviewer  Name). 

I  am  calling  from  Westat,  a  research  firm  near  Washington,  D.C. 
We  are  conducting  an  important  national  survey  for  the  U.S. 
Government . 


Revised  Version 


Hello,  this  is  (Interviewer  Name). 

I  am  calling  from  Westat,  a  research  firm  near  Washington,  D.C. 
We  are  conducting  a  study  for  the  United  States  Government 
about  people's  plans  for  the  future  and  their  reaction  to  Armed 
Forces  advertising. 

A  second  change  tc  the  screening  interview  was  made  in  response 
to  interviewer  feedback  that  some  parental  respondents  appeared  to  be 
giving  inaccurate  information  on  household  composition  in  order  to 
avoid  participating  in  the  study.  The  household  composition  question 
was  therefore  revised  to  diffuse  the  focus  from  the  specific  popula¬ 
tion  of  interest.  The  change  in  wording  is  shown  below. 
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Original  Version 


Since  the  survey  we  are  conducting  for  the  U.S.  government  is 
concerned  with  the  career  plans  of  young  adults ,  we  need  to 
know  how  many  young  adults  live  in  your  household.  Including 
anyone  away  on  vacation,  away  on  business  or  living  away  at 
school,  how  many  young  people  between  the  ages  of  13  and  24 
live  in  your  household? 


Revised  Version 


We  have  a  few  questions  to  see  if  anyone  in  your  household  will 
be  included  in  this  study.  Including  yourself... 

how  many  people  aged  25  or  older  live  in  your  household? 

how  many  people  between  the  ages  of  13  and  24  live  in  your 
household? 

Interviewer  staffing  changes.  Changes  to  the  interviewer 
staffing  patterns  were  made  at  the  same  time  as  the  screener  modifi¬ 
cations.  Observation  of  interviewing  productivity  showed  variations 
according  to  time  of  day  and  day  of  week.  For  example,  weekday 
mornings  were  less  productive  than  other  times  for  youth  interviews. 
Staffing  levels  were  therefore  adjusted  such  that  the  interviewing 
workload  coincided  with  the  most  productive  periods. 

The  results  of  both  of  these  changes  can  be  seen  by  comparing 
the  response  and  eligibility  rates  of  the  first  and  second  quarters 
of  data  collection.  Table  4  shows  no  major  improvements  between  the 
first  and  second  quarters  in  the  screener  response  rates,  youth 
eligibles  located  and  interviews  completed.  Household  and  youth 
response  rates  for  the  second  quarter  were  312  and  762,  respectively, 
compared  to  842  and  742  obtained  in  the  first  quarter.  Youth  eligi¬ 
bility  ratios  also  remained  essentially  stable  between  the  first  and 
second  quarters,  indicating  that  the  modifications  to  the  screener 
did  not  uncover  any  more  eligibles  than  those  previously  reported. 

It  appeared,  therefore,  that  other  modifications  to  the  survey 
procedures  would  be  necessary  in  order  to  obtain  the  desired  youth 
sample  sizes. 

Eligibility  determination  changes.  At  the  start  of  the  third 
quarter  (April  1987),  another  change  was  made  in  order  to  improve  the 
youth  eligibility  rates.  During  the  first  two  quarters  of  data 
collection,  youth  were  excluded  from  the  main  interview  if  the  house¬ 
hold  screener  respondent  refused  to  answer  or  did  not  know  the  answer 
to  any  questions  that  determined  eligibility  for  the  survey.  Since 
many  screener  respondents  could  not  answer  questions  regarding  the 
education  and  military  experience  of  all  youth  in  the  household,  it 
was  possible  that  some  eligible  youth  were  being  excluded  from 
participation.  For  the  third  quarter,  therefore,  the  final  determi¬ 
nation  of  youth  eligibility  was  postponed  until  the  beginning  of  the 
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main  youth  interview,  when  the  youth  was  asked  all  the  questions  that 
determined  eligibility.  These  questions  were  not  asked  again  during 
the  main  interview  if  the  youth  was  the  household  screener 
respondent . 

These  procedural  changes  did  not  result  in  the  anticipated 
increases  in  the  number  of  eligible  youth.  In  the  first  and  second 
quarters  of  data  collection,  6.1  households  were  needed  to  locate  1 
eligible  youth.  After  the  procedural  changes  were  implemented,  8.6 
and  8.5  households  were  needed  to  locate  1  Primary  Male  Sample  youth 
in  the  first  and  second  quarters,  respectively.  The  eligibility 
ratios  obtained  were  not  much  different.  The  overall  youth  eligibil¬ 
ity  rate  remained  close  to  the  6.1  to  1  ratio  obtained  in  the  first 
two  quarters  (6.0  to  1;  6.3  to  1;  and  6.2  to  one  for  the  third, 
fourth,  and  fifth  quarters,  respectively).  Eligibility  ratios  for 
other  youth  groups  also  showed  no  major  improvements. 

PMS  eligibility  ratios  also  remained  stable.  In  the  third, 
fourth,  and  fifth  quarters,  respectively,  8.4,  8.7,  and  8.8  house¬ 
holds  were  needed  to  locate  1  PMS  youth.  These  ratios  were  very 
similar  to  the  first  and  second  quarter  ratios. 

Increasing  the  number  of  telephone  numbers  per  month.  Since  the 
response  rates  for  both  the  household  and  youth  interviews  were 
relatively  stable  over  time  (see  Tables  4  and  6)  and  were  very 
similar  to  response  rates  obtained  in  comparable  youth  surveys  such 
as  the  Youth  Attitude  Tracking  Survey  we  did  not  feel  that  the  sample 
yield  could  be  improved  substantially  by  further  efforts  to  improve 
response  rates.  In  addition,  it  appeared  that  the  eligibility  ratios 
being  obtained  could  also  be  expected  to  remain  «=*-able  for  the  dura¬ 
tion  of  the  study. 

It  therefore  appeared  prudent  to  enlarge  the  pool  of  telephone 
numbers  from  which  respondents  could  be  obtained.  The  number  of 
telephone  numbers  in  the  monthly  sample  was  increased  at  the  start  of 
the  third  quarter  by  2,000,  from  10,922  to  12,922. 

The  increase  in  monthly  telephone  numbers  contacted  resulted  in 
the  expected  gains  in  obtained  sample  sizes,  although  the  gains  were 
not  as  yet  sufficient  to  reach  the  monthly  sample  sizes  desired. 
Therefore,  the  process  of  increasing  the  number  of  telephone  numbers 
contacted  monthly  continued  incrementally  under  close  scrutiny  for 
yield  and  cost  implications.  Increases  in  the  sample  of  telephone 
numbers  were  managed  without  corresponding  cost  increases  by  continu¬ 
ing  to  adjust  interviewer  staffing  patterns  to  optimize  productivity. 

In  June  1987,  the  monthly  sample  of  telephone  numbers  was  again 
increased  by  2,000,  and  an  additional  1,000  numbers  were  added  in 
July  and  again  in  October,  bringing  the  monthly  sample  of  telephone 
numbers  to  16,922.  At  this  level,  it  was  felt  that  the  annual  sample 
size  targets  could  be  obtained  for  the  year  starting  in  the  fall  of 
1987. 
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As  expected,  growing  experience  with  the  management  of  a  monthly 
sampling  and  data  collection  effort  showed  steady  improvement  in  data 
collection  yields  as  time  progressed.  Shortfalls  experienced  at  the 
start  of  the  tracking  survey  effort  in  the  fall  of  1986  gradually 
decreased  so  that  sample  size  targets  were  being  met  by  the  fall  of 
1987.  These  improvements  were  largely  due  to  the  increase  in  the 
sample  of  monthly  telephone  numbers  contacted  each  month.  The  pool 
was  gradually  increased  without  a  corresponding  increase  in  inter¬ 
viewer  hours,  although  staffing  patterns  were  carefully  monitored  by 
day  of  the  week  and  time  of  day. 

As  discussed  previously,  various  other  measures  taken  to 
increase  the  obtained  youth  sample  size  did  not  result  in  major 
improvements . 

Future  surveys  of  similar  populations  can  be  expected  to  experi¬ 
ence  similar  eligibility  and  response  rates  found  in  the  ACOMS 
survey.  Fairly  large  differences  were  obtained  between  the  estimates 
derived  largely  from  1980  Census  statistics  (projected  to  1986), 
suggesting  that  real  population  shifts  may  have  occurred  in  the 
interim.  In  addition,  it  is  possible  that  telephone  coverage  may 
also  have  declined  somewhat  during  this  period. 

By  the  fall  of  1987,  the  data  collection  effort  had  reached  a 
stable  state,  with  expectations  for  meeting  most  sample  size  goals  in 
the  upcoming  school  year.  Table  7  shows  the  sample  size  targets  and 
data  collection  projections  for  school  year  1987/88,  using  the 
distributions  of  youth  obtained  in  the  fall  quarter.  As  noted 
earlier,  sample  size  targets  for  all  major  sample  groups  were 
expected  to  be  achieved  or  exceeded  in  the  upcoming  year. 

Parental  Sample 

The  parental  sample  consisted  of  fathers  and  mothers  of  16-  to 
20-year-old  Primary  Sample  youth.  One  parent  in  each  household  with 
PMS  or  PFS  youth  was  selected  for  interview.  As  described  more  fully 
in  Chapter  -  the  gender  of  the  parent  to  be  interviewed  was  pre¬ 
determined  t  the  household  level  by  the  CATI  program.  In  cases 
where  multiple  parental  figures  of  the  appropriate  gender  were  avail¬ 
able  (e.g.,  stepfathers  and  fathers),  the  designation  of  the  specific 
parental  figure  to  be  interviewed  was  determined  by  the  youth 
respondent. 

Table  8  shows  the  parental  sample  sizes  and  response  rates  by 
quarter.  A  total  of  4,144  parents  of  primary  sample  youth  were 
interviewed.  Although  half  of  the  households  were  designated  for 
fathers  and  half  for  mothers,  response  rate  differences  between  the 
fathers  (54. 5X)  and  mothers  (68. 9X),  resulted  in  a  greater  number  of 
mothers  interviewed  (Table  8).  It  should  be  noted  that  the  parental 
response  rate  reported  is  a  very  conservative  estimate  since  the 
denominator  includes  parents  who  could  not  be  located  for  various 
reasons  (e.g.,  diseased  parents,  parents  for  whom  no  complete  names 
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Data  Collection  Projections  for  the  1987/88  School  Year  (SY  87/88) 
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Completed  Interviews  +  refusals  +  unfinalized  cases  +  nonlocatables 


and/or  telephone  numbers  were  obtained  from  the  youth  respondent) . 
Non-locatable  parents  consisted  of  approximately  20X  of  the  parental 
sample  identified. 

Relatively  few  changes  were  made  to  the  procedures  for  collect¬ 
ing  data  from  the  parental  sample.  After  the  first  three  quarters, 
efforts  were  made  to  improve  the  parental  response  rates  by  providing 
special  interviewer  training.  Some  youth  were  perceived  as  perform¬ 
ing  a  "gatekeeper"  function  for  their  parents,  refusing  outright  to 
allow  contact  with  their  parents  or  refusing  to  provide  adequate 
information  to  enable  parental  contact.  Interviewers  were  trained  in 
techniques  of  converting  the  youth's  refusals  by  explaining  the 
importance  of  allowing  the  parent  to  make  his/her  own  decision  to  be 
interviewed .  Some  youth  were  also  concerned  about  the  confidential¬ 
ity  of  the  information  they  provided.  Interviewers  were  able  to 
obtain  additional  parental  names  after  assuring  the  youth  of  the 
confidentiality  of  their  responses  and  explaining  the  content  of  the 
parental  questionnaire.  However,  no  major  response  rate  improvements 
appeared  to  result  from  these  efforts. 
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3.  ACOMS  SURVEY  INSTRUMENTS 

Bruce  F.  Allen,  Veronica  F.  Nieva,  and  Gregory  H.  Gaertner 

The  Army  Communications  Objectives  Measurement  System  (ACOMS) 
includes  three  survey  instruments:  (a)  the  household  screener, 

(b)  the  youth  questionnaire,  and  (c)  the  parent  questionnaire.  The 
content  and  structure  of  each  are  described  in  this  chapter.  In 
addition,  major  changes  made  to  each  instrument  are  also  discussed 

The  Household  Screener 


The  household  screener  was  a  three-minute  instrument  designed  to 
locate  eligible  respondents  for  the  full  ACOMS  interview.  Using  the 
Random  Digit  Dialing  (RDD)  method  of  sample  selection,  telephone 
numbers  were  contacted  at  random.  The  screener  interview  was  used  to 
identify  and  eliminate  nonhousehold  telephone  numbers  such  as  busi¬ 
nesses  and  institutional  telephone  numbers  as  well  as  households  that 
did  not  have  youth  who  fulfilled  ACOMS  eligibility  requirements. 

The  screener  interview  was  conducted  with  any  knowledgeable 
household  member  who  was  obviously  not  a  young  child.  If  the 
screener  respondent  did  not  know  key  information  such  as  the  number 
of  youth  in  the  household,  the  respondent  was  asked  to  call  another 
household  member  to  the  phone.  If  no  one  else  was  available,  an 
appointment  was  made  to  call  the  household  at  a  time  when  a  knowl¬ 
edgeable  household  member  was  likely  to  be  home.  Screener  respon¬ 
dents  were  often  youth  eligible  for  the  ACOMS  interview.  In  such 
cases,  after  completion  of  the  screener  instrument,  the  respondent 
was  guided  directly  into  the  main  youth  interview.  Otherwise, 
appointments  were  made  for  each  youth  eligible  for  the  main 
interview. 

The  screener  began  with  verification  and  identification  of  the 
telephone  number.  The  interviewer  verified  the  telephone  number, 
identified  him/herself,  and  asked  several  questions  designed  to 
screen  out  business  telephone  numbers.  The  screener  then  inquired 
about  the  number  of  youth  between  16  and  24  years  of  age  who  cur¬ 
rently  live  in  the  household.  If  the  telephone  number  was  dialed 
incorrectly,  if  the  number  was  a  business,  or  if  the  household  did 
not  contain  an  age -eligible  youth,  the  interviewer  read  a  brief 
statement  of  thanks  for  the  respondent' s  time  and  terminated  the 
interview. 

In  households  with  age-eligible  youth,  the  interviewer  read  a 
Privacy  Act  statement  that  guaranteed  confidentiality  of  responses 
and  notified  the  respondent  that  participation  was  voluntary.  In 
addition,  a  more  detailed  description  of  the  purpose  and  sponsor  of 
the  study  was  presented.  Following  this  explanation,  information  on 
the  names  and  ages  of  the  age-eligible  youth  was  solicited.  In  all 
households,  this  information  was  requested  for  all  age-eligible 
males.  Parallel  information  for  age-eligible  females  was  only  col¬ 
lected  in  a  random  20%  of  all  households  which  were  pre-designated  as 
potentially  available  for  female  interviews. 
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Once  all  desired  youth  were  enumerated,  the  interviewer  asked  a 
series  of  questions  on  the  youth' s  educational  attainment  and  mili¬ 
tary  service  record.  Answers  to  these  questions  were  used  to  assess 
the  youth's  eligibility  for  participation  in  the  ACOMS  survey.  Ques¬ 
tions  were  also  asked  to  clarify  whether  the  youth  was  currently 
living  in  that  household  or  in  a  college  dormitory  or  boarding 
school.  In  the  latter  case,  a  phone  number  where  the  youth  could  be 
reached  during  the  school  year  was  obtained.  The  race,  last  name, 
and  county  of  residence  for  each  desired  youth  enumerated  were  then 
collected. 

Elimination  of  13-  to  15-Year-Old  Tracking  Information 

During  the  first  two  quarters  of  ACOMS  data  collection,  tracking 
information  was  collected  for  13-  to  15 -year- olds  identified  during 
the  screening  interview.  The  information  was  originally  collected  as 
a  cost- saving  mechanism,  accompanying  an  earlier  iteration  of  the 
sample  design.  It  was  proposed  then  that  longitudinal  and  cross- 
sectional  samples  could  possibly  be  combined  as  a  way  of  reducing 
sample  size  requirements.  Under  such  conditions,  it  would  have  been 
possible  to  save  screening  costs  substantially  by  "pre- identifying" 
youth  who  would  enter  the  sample  in  subsequent  years . 

However,  the  final  sampling  design  calling  for  a  "fresh"  cross- 
sectional  sample  each  year  minimized  the  potential  savings  from  the 
"pre -identified"  youth.  Later  design  changes  that  limited  the 
longitudinal  sample  to  16-  to  20-year-old  youth  further  diminished 
the  utility  of  collecting  information  on  the  13-  to  15 -year -old 
youth.  These  considerations,  combined  with  the  Army's  concern  about 
potential  public  sensitivity  to  collecting  information  on  youth  below 
the  recruiting  market  age -boundaries  led  to  a  decision  to  discontinue 
the  enumeration  and  tracking  of  13-  to  15-year-old  youth  beginning 
July  1,  1987. 


The  Youth  Interview 


The  Hierarchy  of  Effects  Model 

The  main  Army  Communications  Objectives  Measurement  System 
(ACOMS)  survey  instruments  consisted  of  youth  and  parental  question¬ 
naires  based  on  a  modified  hierarchy  of  effects  model  (Fishbein  & 
Azjen,  1975)  of  the  advertising  process  (see  Figure  1.).  Implicit 
in  the  model  was  the  notion  that  (a)  the  consumer  gains  awareness  of 
a  product  through  advertising  messages,  which  result  in  (b)  beliefs 
about  the  product,  in  interaction  with  the  consumer's  evaluation  of 
his/her  need  for  the  product,  which  in  turn  affect  (c)  the  respon¬ 
dent's  attitude  toward  the  product,  and  (d)  subsequent  intentions  and 
actions  toward  the  product.  The  model  therefore  suggested  the  major 
types  of  criteria  by  which  ACOMS  assessed  advertising  effectiveness: 

(1)  Awareness  (or  recall)  of  Army  advertisements; 

(2)  Positive  beliefs  or  perceptions  about  Army  attributes; 
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ure  1 .  Expanded  Flshbein-Azjen  hierarchy  of  effects  node  l  for  ACOMS 


(3)  Positive  attitudes  toward  Army  enlistment,  intentions  to 
enlist;  and 

(4)  Actual  enlistment -related  behaviors. 

The  model  also  posited  the  impact  of  social  influence  on  the 
youth's  intentions  to  enlist.  Social  norms,  in  general,  and  the 
youth's  parents,  in  particular,  were  seen  as  significant  factors  in 
the  youth' s  attraction  to  the  Army . 

Parental  attitudes  and  behaviors  were  hypothesized  to  be 
affected  by  Army  advertising,  in  a  process  that  paralleled  that  of 
the  youth.  According  to  the  model,  parents,  like  the  youth,  became 
aware  of  the  Army  through  its  advertising.  If  effective,  such  adver¬ 
tising  changes  their  beliefs  and  attitudes  towards  the  Army  in  a  more 
positive  direction.  Positive  parental  attitudes  were  then  expected 
to  be  manifested  in  parental  behaviors  intended  to  influence  the 
youth's  intention  to  enlist. 

In  the  current  context,  the  hierarchy  of  effects  model  suggested 
important  measures  for  consideration  in  measuring  the  effects  of  Army 
advertising.  Respondent  demographics  (e.g.,  region,  race  and  ethnic¬ 
ity,  and  sex),  life  stage  (e.g.,  age,  employment  and  marital  status, 
income  or  income  of  household,  and  education),  media  habits  and  expo¬ 
sure  were  postulated  to  affect  respondent  awareness  of  the  Army 
gained  through  exposure  to  Army  advertising.  This  level  of  knowledge 
was  hypothesized  to  affect  respondent  beliefs  about  advertising 
messages  such  as  attributes  of  the  Army.  In  Fishbein  and  Azjen's 
model  as  adapted,  these  beliefs  took  the  form  of  agreement  that  a 
given  attribute  was  offered  by  the  Army  (or  component,  other  service, 
college  or  civilian  job). 

Corresponding  to  these  beliefs  were  evaluations  of  the 
importance  of  these  attributes  to  the  respondent.  The  youth's  atti¬ 
tudes  toward  enlistment  were  hypothesized  to  be  the  product  of 
beliefs  about  what  the  Army  offers  and  the  corresponding  personal 
importance  of  those  attributes.  Thus,  attitudes  would  be  positive 
when  the  respondent  believed  that  positively  evaluated  outcomes 
relating  to  military  service  were  likely.  They  would  be  indifferent 
when  perceived  outcomes  were  not  seen  as  important,  or  when  important 
positive  outcomes  seen  as  only  moderately  likely.  Finally,  attitudes 
were  expected  to  be  negative  when  negatively  evaluated  outcomes  were 
seen  as  likely. 

These  attitudes  were  hypothesized,  in  turn,  to  affect  behavioral 
intentions  including  propensity  toward  the  Army,  which  then  were 
expected  to  affect  behaviors.  It  is  important  to  note  that  these 
actions  may  include  enlistment,  but  are  also  likely  to  include  inter¬ 
mediate  actions  like  seeking  information,  asking  for  advice,  seeing  a 
recruiter,  and  so  on.  As  the  model  suggests,  these  intermediate 
actions  could  feed  back  on  beliefs  and  evaluations. 
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The  model  also  included  the  effects  of  social  (especially 
parental)  influence  on  the  decision  process.  Parents  of  a  subset  of 
ACOMS  respondents  were  interviewed  on  their  exposure  to  Army  adver¬ 
tising,  and  beliefs  and  evaluations  of  Army  attributes.  The  parents 
were  also  asked  preferences  for  the  target  youth' s  future  and  how  the 
parents  attempted  to  influence  the  youth's  planning. 

Youth  Interview  Structure 


The  youth  interviews,  which  averaged  about  30  minutes  in  length, 
provide  the  major  measures  of  the  dynamics  and  effects  of  Army  adver¬ 
tising  available  in  ACOMS.  In  accordance  with  the  conceptual  model 
which  has  guided  the  development  of  ACOMS,  the  youth  interview  was 
divided  into  14  topical  modules.  In  order  to  achieve  a  30-minute 
average  interview  while  ensuring  that  all  necessary  information  is 
obtained,  a  complex  questionnaire  structure  was  developed.  The  14 
questionnaire  modules  were  divided  into  8  "core"  modules  which  were 
asked  of  all  respondents,  and  6  "noncore"  modules  which  were  asked 
either  of  a  randomly  selected  subset  of  respondents,  or  of  respon¬ 
dents  with  defined  characteristics  determined  earlier  in  the 
interview. 

Core  modules .  The  core  modules  were: 

(1)  Education  and  Employment:  elicited  employment  history  and 
measures  of  course -content  and  school  performance  useful 
for  assessing  quality. 

(2)  Intentions  and  Propensity:  asked  for  the  respondent's 
plans  for  the  next  few  years,  constructed  to  parallel  and 
supplement  measures  of  Army  propensity  in  the  Youth 
Attitude  Tracking  Study  II  (YATS  II)  (Research  Triangle 
Institute,  1985). 

(3)  Behaviors:  elicited  information  on  the  respondent's 
activities  relative  to  enlistment,  employment  and/or 
college  enrollment. 

(4)  Importance  of  Attributes:  assessed  the  importance  to  the 
respondent  of  attributes  defined  by  the  Army's  communica¬ 
tions  objectives.  These  items  corresponded  to  the 
evaluation  component  of  the  Hierarchy  of  Effects  model. 

(5)  Knowledge -Recall:  asked  for  unaided  and  aided  recall  of 
Army  (by  component)  and  other  service  advertising,  pre¬ 
sented  in  random  order.  The  respondent  was  also  asked 
where  the  advertising  was  seen  or  heard,  what  its  main 
message  was,  and  whether  he/she  believed  and/or  liked  it. 

(6)  Attitude  toward  Army  Advertising:  ascertained  how  much  the 
youth  liked  and  believed  the  advertisements  he/she  had  seen 
or  heard. 
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(7)  Perceptions/Beliefs:  asked  whether  the  Army  (by  compo¬ 
nent;,  other  services,  military  service  in  general,  college 
and  civilian  employment  offered  the  attributes  defined  by 
the  Army's  communications  objectives  presented  in  random 
order.  These  items  corresponded  to  beliefs  in  the 
Hierarchy  of  Effects  model. 

(8)  Demographics:  elicited  information  on  respondent's  ethnic¬ 
ity,  marital  status,  Social  Security  number,  socioeconomic 
background,  and  current  address.  Demographic  data  was  used 
for  weighting  purposes,  and  for  analytic  comparisons  of 
subgroups . 

It  should  be  noted  that  one  of  the  core  modules,  Percep¬ 
tions/Beliefs  about  the  Army  and  other  career  options,  was  divided 
into  sections  that  were  assigned  to  respondent  subgroups  differenti¬ 
ated  by  educational  status  and  plans.  These  allocation  strategies 
are  discussed  in  further  detail  later  in  this  chapter. 

Non-core  modules.  Non-core  modules  were  asked  only  of 
particular  respondents.  The  non-core  modules  were: 

(1)  Media  Habits:  elicited  information  on  the  amounts  of 
television,  radio,  and  print  material  the  respondent  is 
regularly  exposed  to,  and  his/her  favorite  programs  and 
print  vehicles  (asked  only  of  a  randomly  selected  half  of 
all  youth  respondents). 

(2)  Knowledge -Slogan  Recognition:  asked  whether  the  respondent 
could  identify  slogans  utilized  in  Army,  other  services  and 
joint-service  advertising  presented  in  random  order  (asked 
only  of  a  randomly  selected  half  of  all  youth  respondents). 

(3)  Knowledge -Awareness :  asked  for  the  respondent's  level  of 
knowledge  concerning  Army  offers  (asked  only  of  a  randomly 
selected  half  of  all  youth  respondents). 

(4)  Parental -Location  Information:  elicited  information 
required  to  contact  parents  (asked  only  of  parental -linked 
youth  sample) . 

(5)  Social  Influences:  asked  for  the  respondent's  assessments 
of  the  attitudes  of  friends,  parents  and  others  toward 
enlistment  (asked  only  of  the  parental-linked  youth 
sample) . 

(6)  Tracking  Information:  elicited  information  required  to 
trace  youth  selected  for  inclusion  in  the  longitudinal 
sample,  including  anticipated  changes  in  telephone  number, 
names  and  phone  numbers  of  employer  and  two  others  likely 
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to  know  respondent's  whereabouts  (asked  only  of  longitudi¬ 
nal  sample) . 

The  Slogan  Recognition,  Knowledge -Awareness ,  and  Media  Habits 
modules  were  asked  of  only  a  randomly  selected  half  of  all  youth 
respondents  and  each  possible  pair  of  modules  was  asked  of  one-sixth 
of  the  sample.  Slogan  Recognition,  Knowledge -Awareness  and  Media 
Habits  modules  were  distributed  among  respondents  as  shown  in 
Table  9 . 

Respondents  were  randomly  assigned  to  each  of  these  three 
modules  with  a  probability  of  .333.  Half  were  not  assigned  a  second 
module  from  this  group  of  three.  The  remaining  half  received  one  of 
the  two  remaining  modules  with  equal  probability. 


Table  9 

Module  Rotation  Plan:  Percentage  of  Youth  Respondents  Assigned 
to  Modules 


Media 

Habits 

Second  Module 

Slogan 

Recognition 

Knowledge  - 
Awareness 

None 

Percentage 

Media  Habits 

O.OZ 

8.3Z 

8.3Z 

16. 7Z 

33. 3Z 

Slogan 

Recognition 

8.3Z 

O.OZ 

8.3Z 

16. 7Z 

33. 3Z 

Knowledge- 

Awareness 

8.3Z 

8.3Z 

O.OZ 

16. 7Z 

33. 3Z 

Percentage 

16. 7Z 

16. 7Z 

16. 7Z 

50Z 

Allocation  of  the  Perceptions/Beliefs  sections.  The  Percep¬ 
tions/Beliefs  module  contained  questions  about  those  attributes  which 
constituted  the  major  copy  points,  or  communications  objectives,  of 
Army  advertising.  These  communications  objectives  were  developed  in 
an  iterative  process  that  involved  group  and  individual  discussions 
with  Special  Advisory  Group  (SAG)  members  and  other  Army 
representatives . 


2 

Originally,  plans  included  a  longitudinal  sub -sample  of  youth,  who 
were  to  e  reinterviewed  at  annual  intervals.  This  subsample  was 
later  dropped  from  the  project. 
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ACOKS  j.espondents  were  asked  whether  these  attributes  were  | 

descriptive  of  10  referents  (active  Army,  Reserve  Officers'  Training 
Corps  (ROTC) ,  Army  National  Guard  (ARNG) ,  U.S.  Army  Reserve  (USAR) , 

Navy,  Marines,  Air  Force,  military  service  generally,  college,  and 
work)  that  may  be  perceived  by  the  youth  as  future  career  options. 

Table  10  shows  the  attributes  asked  about  each  of  the  service  and 

career  referents .  j 

The  attributes  listed  for  each  referent  was  considered  a 
"section"  in  the  Perceptions  module.  Since  asking  any  single  indi¬ 
vidual  to  respond  to  all  10  sections  would  have  been  an  intolerable 
burden  to  the  respondent,  a  plan  to  allocate  respondents  to  Percep¬ 
tions  sections  was  devised  in  collaboration  with  the  SAG.  j 

The  allocation  scheme  was  intended  to  reflect  the  market  priori¬ 
ties  of  each  of  the  Army  components .  Respondents  were  divided  into 
six  subsamples: 

(1)  Those  who  have  completed  three  or  more  years  of  college;  j 

(2)  Respondents  currently  enrolled  in  first  or  second  year  of 
college; 

(3)  High  school  students  intending  to  go  to  college, 

(4)  High  school  students  not  intending  to  go  to  college; 

(5)  High  school  diploma  graduates  not  currently  enrolled  in 
college;  and 

(6)  High  school  noncompleters.  I 

The  specific  rules  of  the  allocation  scheme  are  described  below: 

(1)  All  respondents  were  asked  about  two  components,  usually 
active  Army  and  one  other.  However,  one  group  (college- 

bound  high  school  students)  was  asked  three  modules.  j 

(2)  All  Primary  Male  Analytic  Sample/Primary  Female  Analytic 
Sample  (PMAS/PFAS)  eligibles  were  asked  the  active  Army 
perceptions  questions  (Groups  2  through  5). 

(3)  Those  attaining  three  or  more  years  of  college  (Group  1) 
were  asked  two  sets  of  questions,  those  pertaining  to  ROTC 
and  one  other  component,  branch,  college,  or  civilian  job. 

(4)  College  freshmen  and  sophomores  (Group  2)  were  asked  two 
modules,  either  active  Array,  or  ROTC,  and  one  other.  One- 
third  of  Group  2  was  asked  active  Army,  and  ROTC,  one -third 
active  Army  and  one  other  module,  and  one -third  ROTC,  and 
one  other  module. 
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Allocation  of  Q87-2  Perceptions  to  Service  and  Career  Options  (continued) 
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Allocation  of  Q87-2  Perceptions  to  Service  and  Career  Options  (continued) 
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Total  Attributes 


(5)  College-bound  high  school  students  (Group  3)  were  asked 
three  modules,  active  Army,  ROTC,  and  one  other. 

(6)  Noncollege-bound  high  school  students  and  high  school 
graduates  not  currently  enrolled  (Groups  4  and  5, 
respectively)  were  asked  active  Army  and  one  other  set. 

(7)  Secondary  Male  Sample/Secondary  Female  Sample  (SMS/SFS) 
respondents  were  allocated  as  Group  4  and  5 . 

Two  factors  combined  to  determine  the  number  of  youth  who 
actually  received  particular  perceptions  lists.  First,  the  random¬ 
ization  programming  determined  the  percentage  of  each  panel  receiving 
each  list.  Second,  the  sample  composition  had  to  match  the  estimates 
of  actual  sample  composition  made  prior  to  the  beginning  of  data 
collection. 

Table  11  summarizes  the  performance  of  the  random  assignment. 

For  each  subsample,  the  first  two  rows  represent  the  percentage  of 
that  panel  that  was  planned  to  receive  that  service' s  perceptions 
list,  and  the  actual  percentage  of  the  panel  that  received  the  list. 
For  example,  67%  of  all  freshmen  and  sophomores  were  expected  to 
receive  the  active  Army  perceptions  list,  while  68%  actually  received 
the  list.  The  small  differences  obtained  between  planned  and  actual 
assignment  for  all  panel-service  list  combinations  indicates  that  the 
random  assignment  programming  performed  well. 

Table  12  presents  the  expected  and  actual  PMS  subsample  composi¬ 
tions.  Estimates  of  expected  PMS  subsample  compositions  were  based 
upon  the  Current  Population  Survey  and  the  Fall  1985  Youth  Attitude 
Tracking  Survey  II  (YATS  II) .  We  assumed  that  about  10%  of  all  PMS 
respondents  would  have  completed  the  third  year  of  college  or  more, 
24%  would  be  freshmen  and  sophomores,  11%  of  high  school  students 
would  be  college -bound,  13%  of  high  school  students  would  be 
noncollege -bound,  and  that  42%  of  high  school  graduates  would  not  be 
enrolled  in  college.  We  expected,  for  example,  about  970  sets  of 
responses  to  the  ROTC  questions  annually  from  those  in  their  third 
year  of  college  or  more,  and  so  on. 

However,  as  Table  11  demonstrates,  the  actual  PMS  subsample 
composition  was  quite  different  than  expected.  Major  differences  in 
expected  versus  actual  distributions  were: 

(1)  Fewer  college  freshmen  and  sophomores  were  obtained  than 
expected; 

(2)  More  college-oriented  high  school  students  were  obtained 
than  expected;  and  conversely,  fewer  work-oriented  high 
school  students  were  obtained  than  expected;  and 

(3)  Fewer  high  school  graduates  not  currently  enrolled  were 
obtained  than  expected. 


Performance  of  Randomized  Assignment  of  Subsamples  to  Perceptions  Attributes  Planned  Versus  Actual 


w 

3 

Vi 

4) 

c 

^4 

y 

Vi 

3 

0 

0 

3 

z 

u 

u 

c 

C/5 

u 

CO 

•o 

Vi 

4) 

c 

.* 

c 

4) 

Vs 

4) 

•a 

0 

4) 

3 

:* 

Vs 

vs 

O 

C/3 

C/5 

X 

CO 

•3 

o 

U 

0 

4) 

CO 

C 

00 

C 

y 

y 

4) 

£  -a 

4) 

00 

3 

H 

Vs 

vs 

0 

y 

X 

C/3 

o 

C/5 

o  o 
o  o 


o  o 
o  o 


o  o 
o  o 


*-S  «-S  CM  sM 


^4  ©  HN  »-4  1-1  *-S  CM 


«-*©  *-H  o  *-l  O  fH<M 


r-S  ^  *-S  O 


©  00  r-.  \o 

©  o>  cs 


r*  vO 
CM  CM 


00  CO  CO  03  CO  On 


r**  ao  co 


X  c  y 
o  y  u 
cn  fl  o 
£  "0  B 

x  w  c  o 
M  y  «  X 

-S  Vs  CL 

X  Us  O 

CO 


X  3  <0  OC 
U  4)  Vs  y 
VJ  *o  «0  H 

C  0)  ^ 

x  y  >•  o 

CO  vs  o 
^  U  + 
s  <  r>  <« 
o 


N  00  O  00  O'  00  ON  00  00  COCO  00  00  COON  00  00  00  00 

VO  VO  VO  VO 


©  © 


©  00 
©  O' 


CM  «-l  CM  C"1 


CM  CM  CN  CO 


vs 

u 

Vi 

vs 

vs 

4J 

vs 

VS 

U 

VS 

VJ 

VJ 

vs 

VS 

vs 

VS 

u 

u 

VS 

VJ 

c 

e 

c 

c 

C 

c 

e 

c 

c 

c 

c 

c  . 

c 

c 

c 

c 

C 

c 

c 

c 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

vs 

vs 

U 

u 

u 

w 

u 

u 

Vs 

u 

Vs 

u 

Vs 

u 

Vs 

Vs 

Vs 

Vs 

Vs 

Vs 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

V 

y 

y 

y 

y 

y 

Os 

04 

Oe 

0- 

04 

Os 

0s 

0s 

Os 

o* 

04 

04 

Os 

Os 

Os 

Os 

Os 

Os 

Os 

Os 

•3 

H 

•3 

H 

3 

3 

3 

H 

3 

H 

3 

^s 

3 

H 

3 

H 

3 

H 

y 

<4 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

c 

3 

C 

3 

c 

3 

c 

3 

C 

3 

C 

3 

C 

3 

c 

3 

C 

3 

c 

3 

c 

vj 

C 

u 

c 

u 

c 

4J 

C 

u 

c 

u 

C 

vs 

c 

4J 

c 

u 

c 

u 

y 

u 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

y 

04 

< 

04 

< 

^4 

04 

< 

^S 

Os 

< 

^S 

Os 

< 

H 

04 

< 

rL 

Os 

< 

#■4 

Os 

< 

rL 

Os 

< 

^■s 

Os 

< 

CO 

y 

•  • 

•  • 

•  • 

VJ 

CO 

to 

CO 

3 

y 

y 

•  * 

•  • 

•  • 

•  • 

•  ♦ 

y 

X 

Vi 

*  ■ 

Vi 

w 

co 

co 

CO 

co 

Vi 

3 

to 

3 

y 

y 

y 

y 

y 

3 

Vs 

X 

y 

X 

VJ 

u 

vs 

vj 

vs 

X 

Vi 

•^4 

Vi 

■*s 

3 

3 

3 

3 

3 

««S 

Vi 

Vs 

3 

Vs 

X 

X 

X 

X 

X 

Vs 

< 

Vi 

X 

Vi 

■*S 

•mS 

•h 

••s 

«»S 

Vi 

Vi 

V 

Vi 

Vs 

Vs 

Vs 

Vs 

Vs 

Vi 

y 

< 

Vs 

< 

VJ 

u 

VJ 

vj 

vs 

< 

y 

Vi 

4J 

u 

u 

4J 

vs 

Vs 

y 

Vi 

y 

< 

< 

< 

< 

< 

y 

o 

CO 

< 

y 

c 

hs 

y 

*>s 

>* 

O 

O 

oc 

>% 

H 

> 

a 

H 

< 

y 

Vs 

Vs 

H 

Vs 

Vs 

Vs 

O 

52 

V) 

y 

y 

eS 

o 

0 

y 

< 

04 

5! 

© 

z 

£ 

< 

o 

3 

C/3 

42 


4 


I 


i 


* 


n 


I 

•s 


1 

% 


1 

1 


*4 


w 


Table  12 


Estimated  Versus  Actual  Subsample  Composition 


Estimated 

Subsample 

Composition 

Ac  tual* 

Subsample 

Composition 

N 

X  of 
Sample 

N 

Z  of 

Sample 

Attained  3+  Years 
of  College 

970 

10Z 

962 

10Z 

Freshmen  & 

Sophomores 

2,338 

25X 

1,570 

17Z 

H.S.  Students 

College -Oriented 

1,035 

11X 

2,987 

32Z 

H.S.  Students 
Work-Oriented 

1,265 

14Z 

776 

8Z 

H.S.  Grads  Not 
Enrolled 

3,700 

40Z 

3,091 

33Z 

Total 

9,300 

100Z 

9,386 

100Z 

♦Actual  panel  composition  of  reports;  data  collected  October,  1986 
through  December,  1987. 
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Approximately  10%  of  total  PMS  were  college  juniors  and  seniors, 
which  corresponded  closely  with  expectations.  However,  the  number  of 
college  freshmen  and  sophomores  interviewed  fell  short  of  expecta¬ 
tions.  While  our  plans  expected  about  a  quarter  of  the  PMS  sample  to 
be  college  freshmen  and  sophomores,  these  groups  actually  comprised 
only  17%  of  total  PMS  interviewed.  The  difference  was  due  largely  to 
the  difficulty  in  locating  and  interviewing  these  youth.  College 
freshmen  and  sophomores  were  located  through  their  nonschool 
households  (i.e.,  parental  households  or  independent  households). 

Many  youth  located  at  their  parental  households  were  contacted  at 
their  phone  number  at  the  school  housing.  The  response  rate  for 
youth  in  dormitories  was  17%  lower  than  the  response  rate  for  youth 
living  in  the  parental  home  or  in  independent  households . 

The  proportion  of  college-oriented  high  school  students  was 
almost  three  times  as  high  as  expected.  (The  distinction  between 
college-  and  work-oriented  students  was  determined  by  the  response  to 
the  question  "How  likely  is  it  that  you  will  be  going  to  college  in 
the  next  few  years?")  This  was  due  to  two  factors.  First,  high 
school  students  were  easier  to  locate  and  interview  compared  to 
college  students  and  non-enrolled  high  school  graduates  since  the 
majority  lived  in  households  with  their  parents.  Secondly,  most  high 
school  youth  identified  themselves  as  college-oriented,  thus  increas¬ 
ing  the  proportion  of  college-oriented  high  school  students  and 
decreasing  the  proportion  of  work-oriented  high  school  students.  The 
proportion  of  work-oriented  students  was  much  lower  than  expected  (8% 
vs  14%),  which  indicates  that  many  youth  who  would  not  actually  go  to 
college  identify  themselves  as  college -oriented  during  high  school. 

The  proportion  of  high  school  graduates  not  enrolled  in  a  two- 
or  four-year  college  in  PMS  was  also  lower  than  expected  (33%  vs. 
40%).  While  we  do  not  have  specific  data  on  the  subject,  we  suspect 
that  this  shortfall  may  have  been  due  to  telephone  undercoverage 
among  these  youth,  and  the  difficulty  of  scheduling  and  interviewing 
this  mobile  and  busy  population. 

Presentation  of  Perceptions  attributes.  The  Perceptions  module 
contained  lists  of  attributes  for  work,  college,  military  service  in 
general,  the  Army  and  its  components,  and  the  other  services. 
Respondents  were  asked  whether  they  agreed  or  disagreed  with 
attribute  statements  on  a  five  point  scale,  from  disagree  strongly  to 
agree  strongly.  Respondents  could  receive  up  to  three  lists  of 
attributes,  each  list  containing  8  to  14  attributes.  The  maximum 
number  of  attributes  for  a  given  respondent  was  36  attributes. 

During  the  first  quarter,  the  lists  were  administered  sequen¬ 
tially.  For  example,  youth  were  asked  to  respond  to  14  Army 
attributes,  then  14  Air  Force  Attributes,  then  8  ROTC  attributes. 

Each  attribute  list  was  presented  using  a  randomly  selected  attribute 
as  the  starting  point. 

During  the  pretest  and  first  quarter  of  data  collection,  inter¬ 
viewers  reported  that  respondents  became  bored  with  the  presentation 
of  long  attribute  lists  within  each  referent  service.  Data  indicated 


that  most  responses  tended  toward  the  high  end  of  the  scale  with 
little  variance.  It  was  feared  that  the  length  and  monotony  of  the 
lists  resulted  in  a  loss  of  interest  and  fixation  on  a  single 
response  in  order  to  move  through  the  interview  as  fast  as  possible. 
This  potential  "response  set"  threatened  a  number  of  analyses  central 
to  the  mission  of  ACOMS  since  the  attributes  represented  the 
communications  objectives  of  the  Army  and  its  components. 

To  reduce  potential  response  set  bias,  the  Perceptions  module 
was  reprogrammed  to  present  the  attribute  lists  by  attribute  across 
service  rather  than  presenting  the  attribute  lists  bv  service.  For 
example,  during  the  first  quarter  of  data  collection,  respondents 
were  asked,  "Do  you  agree  or  disagree  that  the  Army  offers  a  mentally 
challenging  experience?  Do  you  agree  or  disagree  that  the  Army 
offers  an  experience  you  can  be  proud  of?  Do  you  agree  or  disagree 
that  the  Army  offers..."  When  the  list  of  attributes  was  completed 
with  the  Army  as  the  referent,  the  list  of  attributes  was  asked  again 
for  a  second  service.  Table  13  presents  the  order  of  presentation, 
by  service,  for  14  attributes  used  during  Fall  1986  data  collection. 


Table  13 


Order  of  Perceptions  Module  Attribute  List  Presentation,  Fall  1986 


Service 

Service  1 

List 

Service  2 

Attribute 

1 

1 

15 

Attribute 

2 

2 

16 

Attribute 

3 

3 

17 

Attribute 

14 

14 

28 

Beginning  with  the  second  quarter  of  data  collection,  respon¬ 
dents  were  asked  "Do  you  agree  or  disagree  that  the  Army  offers  a 
mentally  challenging  experience?  How  about  the  (SECOND  SERVICE)?  Do 
you  agree  or  disagree  that  the  Army  offers  an  experience  you  can  be 
proud  of?  How  about  the  (SECOND  SERVICE)?  Do  you  agree  or  disagree 
that  the  Army  offers..."  Table  14  represents  the  order  of  presenta¬ 
tion  by  attribute  used  during  Winter  1987  and  the  remainder  of  ACOMS 
data  collection. 
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Table  14 


Order  of  Perceptions  Module  Attribute  List  Presentation,  Winter  1987 
through  Fall  1987 


Service 

Service  1 

List 

Service  2 

Attribute 

1 

1 

2 

Attribute 

2 

3 

4 

Attribute 

3 

5 

6 

Attribute 

14 

27 

28 

Despite  the  changes  in  presentation  of  attributes,  no  differ¬ 
ences  were  found  between  the  distributions  and  summary  measures  of 
central  tendency  found  in  the  first  and  second  quarters  of  percep¬ 
tions  data.  A  difference  between  these  quarters  would  have  supported 
the  response  bias  hypotheses  for  the  procedure  used  in  the  first 
quarter  (i.e. ,  listing  attributes  within  service).  Further,  factor 
analyses  yielded  a  single  factor  solution  for  both  quarters  of  Per¬ 
ceptions  data.  These  findings  indicated  that  the  observed  patterns 
of  response  were  due  to  a  positive,  undifferentiated  Army  image 
rather  than  response  bias.  See  Wilson  (1987)  for  a  full  discussion 
of  these  analyses  and  findings. 

Recommendations  for  future  instruments.  The  attribute  lists 
contained  within  the  Perceptions  section  represented  the  copy  points 
used  within  Army  and  component  advertising.  The  use  of  these 
attributes  supported  the  Army' s  intention  to  evaluate  the  levels  of 
recall  of  and  agreement  with  the  Army's  communication  objectives. 

The  decision  to  focus  on  these  communication  objectives,  how¬ 
ever,  resulted  in  less  satisfactory  data  for  other  analytic  purposes, 
e.g.,  to  assess  the  "true"  Army  image,  or  brand  differentiation  among 
the  services  and  career  options.  To  meet  these  analytic  objectives 
more  fully,  a  broader  set  of  attributes  would  be  necessary.  Expanded 
sets  of  attributes  should  include  negatively  worded  attributes  and 
attributes  representing  images  not  emphasized  in  advertising.  These 
alternative  attribute  list  constructions  were  discussed  extensively 
during  AGOMS  study  design,  but  were  rejected  in  favor  of  attribute 
constructions  focused  on  the  Army's  communications  objectives. 
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The  Parental  Interview 


Interviews  were  conducted  with  parents  of  16-  to  20-year-old  PMS 
or  PFS  youth,  referred  to  as  the  "parental  linked  sample."  In  house¬ 
holds  which  contained  no  eligible  youth,  no  parental  interview  was 
conducted.  Only  one  parental  interview  was  conducted  within  a  single 
household. 

Identification  of  Parental  Influencers.  and  the  Identification  of 
Head  of  Household 

As  described  in  Chapter  2,  a  random  half  of  the  households  with 
16-20  year  old  PMS  or  PFS  youth  were  pre-designated  for  a  male 
parental  interview,  and  the  other  half  was  pre-designated  for  a 
female  parental  interview. 

During  the  first  quarter  of  data  collection,  the  head  of  house¬ 
hold  was  automatically  interviewed  as  the  parental  influencer  if 
he/she  was  of  the  same  gender  as  that  predesignated  for  the  parent. 
However,  the  pretest  showed  that  this  logic  was  flawed,  since  a  youth 
could  have  several  potential  parental  figures  of  the  selected  gender. 
For  example,  if  a  youth  lives  in  a  household  with  a  mother  and  a 
stepfather,  both  his/her  natural  father  and  stepfather  could  be 
considered  as  the  influential  parent. 

To  resolve  this  issue,  the  identification  of  the  head  of  house¬ 
hold  was  decoupled  from  the  identification  of  the  parental  influ¬ 
encer.  Response  categories  for  questions  regarding  adult  members  of 
the  family  were  reworded  to  clarify  parental  relationship  to  the 
youth.  For  example,  response  categories  were  added  to  questions 
regarding  adult  females  in  the  household  to  allow  for  mothers  or 
stepmothers.  If  the  youth  lived  with  a  stepparent  of  the  selected 
gender,  the  youth  was  asked  to  select  either  the  mother  or  the 
stepmother  as  the  person  who  was  most  influential  when  making  impor¬ 
tant  decisions. 

Structure  of  the  Parental  Interview 

The  parental  interview  contained  eight  modules  parallel  to  the 
youth  instrument  modules.  They  were  asked  in  the  following  order: 

(1)  Parental  Influence:  probed  whether  the  parent  had 
discussed  military  service  with  the  target  youth,  his/her 
expectations  for  the  target  youth,  and  beliefs  that  mili¬ 
tary  service  was  a  good  or  bad  idea  for  most  young  men  and 
women. 

(2)  Importance  of  Attributes:  repeated  the  items  in  the  youth 
version  which  assess  the  importance  of  attributes,  this 
time  asking  the  parent  about  the  importance  of  these 
attributes  to  the  target  youth. 


(3)  Media  Habits:  these  items  were  identical  to  those  asked  of 
the  youth  and  focused  on  the  amounts  of  television,  radio, 
and  print  material  to  which  the  parent  was  regularly 
exposed  and  his/her  favorite  programs  and  print  vehicles. 

(4)  Knowledge -Recall:  these  questions  were  also  identical  to 
those  in  the  youth  questionnaire  and  asked  for  unaided  and 
aided  recall  of  Army  and  other  service  advertising  with 
questions . 

(5)  Attitudes  Toward  Army  Advertising:  addressed  how  much  the 
parent  liked  and  believed  the  advertisements  he/she  had 
heard  or  seen,  using  the  same  items  as  those  in  the  youth 
interview. 

(6)  Perceptions/Beliefs:  probed  with  the  identical  questions 
asked  of  the  target  youth  about  the  extent  to  which  the 
Army  (by  component) ,  other  services ,  military  service  in 
general,  college,  and  civilian  employment  offered  the 
attributes  defined  by  the  Army's  communications  objectives. 

(7)  Knowledge -Awareness :  assessed  the  parent's  knowledge,  as 
in  the  youth  interview,  of  Army  benefits  and  programs. 

(8)  Demographics:  elicited  information  on  the  parent's  ethnic¬ 
ity,  marital  status,  socioeconomic  background  and  military 
experience . 

In  essence ,  the  parental  interview  added  one  module  to  the  youth 
questionnaire  (Parental  Influence) ,  and  dropped  Education  and  Employ¬ 
ment,  Intentions/Propensity,  Behaviors,  Social  Influences,  Slogan 
Recognition,  and  the  Parental  Location  and  Tracking  modules  from  the 
youth  interview.  Except  in  the  Importance  and  Perceptions  modules  in 
which  the  referent  for  the  items  was  changed  to  the  youth  (e.g., 
"Helping  (his/her)  career  development") ,  nearly  all  question-wordings 
were  identical  to  those  in  the  youth  questionnaire.  Further,  where 
random  assignment  to  or  within  modules  was  performed  for  the  youth 
interview,  the  parental  interview  was  assigned  to  the  same  modules 
and  sections.  Thus,  if  the  youth  was  asked  questions  on  perceptions 
of  Army  and  college  attributes,  the  parent  was  also  asked  the  Percep¬ 
tions  questions  for  Army  and  college  attributes.  Further,  if  the 
youth  was  asked  Media  Habits  and  Knowledge -Awareness ,  the  parent  was 
asked  the  same  modules . 

Quarterly  Instrument  Changes 

During  ACOMS  data  collection,  the  survey  instruments  were 
updated  quarterly.  Instrument  modifications  were  conducted  to 
enhance  the  flow  of  the  interview  and  the  ease  of  survey  administra¬ 
tion,  or  to  reflect  changes  in  Army  advertising.  For  example, 
following  the  first  quarter  of  data  collection,  the  question  "Is  this 
person  male  or  female?"  was  modified  to  "Is  this  13-  to  24-year-old 
male  or  female?"  to  eliminate  confusion  regarding  the  referent  of  the 
question.  For  the  second  quarter  instrument,  the  USAR  attribute, 
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'Having  Weekend  Excitement,'  was  reworded  to  read,  'Having  Interest¬ 
ing  and  Exciting  Weekends,'  in  order  to  better  capture  the  intent  of 
the  advertising  message. 


Questionnaire  changes  consisted  of  question  additions,  dele¬ 
tions,  wording  changes,  response  category  changes,  subpopulations 
modifications,  and  changes  in  the  order  of  administration.  Documen¬ 
tation  for  the  ACOMS  questionnaires  and  all  changes  can  be  found  in 
the  ACOMS  Users'  Manual  (Westat,  Inc.,  1988). 
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4.  SURVEY  DATA  COLLECTION  OPERATIONS 
Susan  M.  Englehart 
The  CATI  System 

The  Army  Communications  Objectives  Measurement  Survey  (ACOMS) 
interviews  were  conducted  using  Westat's  proprietary  computer-assisted 
telephone  interview  (CATI)  system.  The  system  consists  of  state-of-the- 
art  software  that  facilitates  the  collection  and  processing  of  complex 
data.  The  CATI  system  was  particularly  appropriate  for  the  ACOMS  survey 
where  the  types  of  information  collected  from  respondents  varied  accord¬ 
ing  to  key  respondent  characteristics. 

The  CATI  system  automated  the  presentation  and  elimination  of  secs 
of  questions  based  on  answers  obtained  earlier  in  the  interview.  Ques¬ 
tion  wording  choices  and  randomized  question  presentation  were  also 
automated.  This  capability  made  the  interviewing  process  considerably 
more  efficient  than  a  hard-copy  instrument  and  resulted  in  significantly 
less  interviewer  error. 

The  CATI  automated  scheduler  program  scheduled  cases  based  on 
algorithms  determined  by  the  project  staff  and  results  of  previous  call 
attempts.  The  result  of  each  call  to  a  telephone  number  (e.g.,  no 
answer,  appointment  made,  or  complete)  was  entered  into  the  system  and 
determined  whether  subsequent  attempts  were  necessary.  A  record  of  all 
attempts  to  each  number  was  maintained  and  could  be  accessed  to  review 
that  number's  call  history. 

CATI ' s  internal  editing  capabilities  also  reduced  the  amount  of 
error  and  time  required  to  process  the  survey  data.  CATI  capabilities 
included  internal  validity  checks  for  value  ranges  and  on-line  con¬ 
sistency  checks  between  related  survey  items.  These  capabilities 
virtually  eliminated  the  need  for  data  retrieval  and  extensive  post¬ 
interview  data  cleaning  and  editing.  Turnaround  time  between  data 
collection  and  analysis  was  therefore  significantly  reduced,  thus 
permitting  quarterly  reports  to  be  produced  in  a  timely  manner. 

Interviewer  Training 


Training  Agenda  and  Format 

The  first  interviewer  training  session  was  held  at  Westat's 
Frederick  Telephone  Research  Center  (TRC)  facility  during  the  period 
October  13-19,  1986.  Seventy-eight  interviewers  attended  this  series  of 
training  sessions.  Additional  training  sessions  were  conducted  in  1987 
during  the  periods  February  1-8,  May  26-31,  and  September  26-October  2 
with  new  groups  of  interviewers.  All  training  sessions  were  conducted 
by  ACOMS  project  members  and  the  Telephone  Research  Center  staff.  The 
trainers  utilized  a  series  of  scripts  designed  to  demonstrate  the 
purpose  of  each  question,  skip  patterns,  consistency  checks,  recording 
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conventions,  and  potential  problems  in  the  ACOMS  interview.  Training 
consisted  of  the  following  eight  components: 


(1)  General  Interviewing  Techniques  (4  hours').  This  session 
introduced  interviewers  to  Wes tat  and  survey  research,  showed 
them  examples  of  types  of  questions  and  recording  conventions, 
taught  basic  ways  to  obtain  accurate  data  through  listening 
and  probing,  and  stressed  methods  for  gaining  respondent  coop¬ 
eration.  Each  interviewer  received  a  manual  that  documented 
the  material  presented  in  the  session. 

(2)  CATI  Training  (4  hours').  This  session  required  trainees  to 
work  with  an  interactive  computer  program  which  instructed 
them  on  the  proper  use  of  the  keyboard  and  the  control  keys 
for  computer-assisted  telephone  interviewing. 

(3)  Study  Overview/Screener  Training/Contact  Procedures  (4  hours’). 
A  brief  introduction  to  the  study,  including  background  infor¬ 
mation  regarding  the  purpose  and  client  was  presented.  The 
remainder  of  the  session  consisted  of  a  detailed  discussion  of 
the  screening  questions  and  the  procedures  for  assigning  the 
appropriate  result  codes  for  all  household  or  non-household 
contacts . 

(4)  Practice  on  Screener  Questionnaire  (2-6  hours').  Prior  to 
continuing  the  next  phase  of  training,  each  interviewer  was 
required  to  spend  2-6  hours  administering  the  screener 
questionnaire  to  respondents  drawn  from  a  "dummy  sample"  in 
the  local  area.  The  number  of  hours  each  interviewer  was 
required  to  practice  the  screener  was  predetermined  by  the 
supervisory  staff  based  on  the  interviewer's  level  of 
experience . 

(5)  Refusal  Avoidance  fl.5  hours)  and  Interactive  Youth  Lecture 
(2.5  hours) .  The  initial  part  of  this  session  consisted  of  a 
review  and  group  discussion  of  the  experiences  and  problems 
encountered  by  the  interviewers  during  their  practice  inter¬ 
views.  A  list  of  questions  frequently  asked  by  respondents 
and  recommended  answers  was  compiled  by  the  project  staff  for 
the  interviewers'  reference.  Techniques  for  avoiding  respon¬ 
dent  refusals  were  discussed. 

The  remainder  of  the  session  was  devoted  to  the  youth  inter¬ 
view  including  a  detailed  discussion  of  the  intent  of  each 
question  in  the  instrument,  ana  instructions  regarding  the 
correct  procedures  for  entering  the  data. 

(6)  Respondent  Selection/Interactive  Parent  Lecture  (4  hours) . 
Since  a  critical  element  of  the  ACOMS  study  was  the  identifi¬ 
cation  and  interviewing  of  the  PMS/PFS  sample,  the  CATI  system 
was  programmed  to  provide  the  interviewer  with  the  ability  to 
select  respondents  according  to  this  priority.  The  specifica¬ 
tions  for  target  youth  and  the  procedures  for  respondent 
selection  were  discussed. 
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Tne  second  part  of  the  session  consisted  of  a  detailed 
demonstration  of  the  parent  questionnaire. 

(7)  Scripted  Practice  (4  hours).  Scripted  interviews  based  on  the 
youth  and  parent  questionnaires  were  presented  during  which 
the  trainer  acted  as  the  respondent  and  interviewers  took 
turns  asking  the  questions.  These  scripts  were  designed  to 
demonstrate  specific  skip  patterns  and  to  reinforce  CATI 
conventions  and  project  definitions  introduced  in  previous 
sessions.  Special  emphasis  was  placed  on  unusual  situations, 
responses,  and  practice  in  answering  respondent  questions. 

(8)  Role  Plavs  (4  hours').  Interviewers  were  divided  into  pairs 
with  one  person  first  assuming  the  role  of  interviewer  and 
then  that  of  the  respondent.  The  "respondent"  was  provided 
with  a  script  designed  to  simulate  an  actual  interview  in 
order  to  permit  both  interviewers  to  practice  the  skills 
learned.  Trainees  were  instructed  to  follow  the  scripts  with¬ 
out  deviation  to  ensure  a  standard  training  experience  and  to 
reinforce  probing  and  problem  handling  techniques  introduced 
during  the  formal  instruction  sessions. 

The  training  facility  was  equipped  with  a  sufficient  number  of 
terminals  to  allow  each  interviewer  to  enter  responses  via  the  keyboard 
during  approximately  half  of  the  training.  A  supervisor  positioned  at  a 
lead  terminal  entered  the  correct  responses  during  training.  This 
terminal  was  connected  to  an  overhead  projector  so  that  the  trainer  and 
interviewers  could  refer  to  an  enlarged  version  of  the  correct  screen 
during  the  training.  All  TRC  supervisory  staff  assigned  to  the  project 
attended  the  training  sessions  and  provided  support  to  the  trainers. 

In  addition  to  the  formal  training,  interviewers  were  requested  to 
spend  4  hours  at  home  reviewing  the  project  manual,  completing  prepared 
exercises,  and  practicing  responses  to  frequently-asked  questions.  The 
total  number  of  training  hours  for  a  new  interviewer  was  34;  the  total 
for  an  experienced  interviewer  was  22. 

Training  Materials 

Written  materials  were  provided  for  the  interviewers  to  use  during 
the  training  itself  and  as  reference  tools  during  the  interview  process. 
The  written  materials  included  the  interviewer  training  manual,  a  refer¬ 
ence  list  of  the  questionnaire  modules,  a  magazine  list,  and  a  list  of 
likely  respondent  questions  coupled  with  preferred  answers. 

The  Interviewer  Training  Manual.  The  ACOMS  Training  Manual 
consisted  of  two  main  parts:  a  general  description  of  the  study,  and 
the  question-by-question  specifications  for  the  survey. 

Figure  2  is  a  copy  of  the  Table  of  Contents  for  the  Training 
Manual.  These  chapters  introduced  the  study's  main  objectives, 
described  sample  selection  procedures ,  discussed  problems  that  might  be 
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encountered  by  interviewers ,  and  provided  a  brief  overview  of  the  survey 
instrument. 

The  introductory  material  was  followed  by  an  appendix  containing 
illustrations  of  all  of  the  screens  in  the  CATI  program.  Accompanying 
the  screen  illustrations  were  explanations  of  when  the  screens  would 
appear  (i.e.,  skip  patterns),  definitions  for  project  purposes  (e.g., 
"household,"  "principal  wage  earner,"  "education  levels,"  etc.),  and 
clarifications  for  questions  that  could  be  misunderstood  or  answered 
ambiguously.  One  example  of  the  latter  was  the  question  "What  is  the 
highest  grade  or  level  of  schooling  that  you  have  completed  and  received 
credit  for?"  Respondents  often  answered  this  question  by  giving  their 
current  educational  status  (e.g.,  "I  am  a  college  sophomore.").  Text 
accompanying  this  question  and  shown  on  the  screen  instructed  interview¬ 
ers  to  attend  to  the  underlined  words  in  the  question  and  to  probe  the 
respondent's  answer  by  verifying  that  he/she  has  completed  and  received 
credit  for  his/her  first  year  of  college.  Additionally,  interviewers 
were  instructed  to  probe  whether  the  respondent  attended  a  2-  or  4-year 
college.  Specification  of  question  intent  was  provided  to  ensure  that 
interviewers  probed  and  coded  respondents'  answers  as  intended  by  the 
questionnaire  designers. 

Additional  training  materials.  In  addition  to  the  training  manual, 
three  other  types  of  reference  materials  were  provided  to  the 
interviewers : 

(1)  Module  List  (Figure  S').  The  16  unique  modules  contained  in 
the  three  questionnaires  (screener,  youth  and  parental 
interviews)  were  listed  by  name  and  reference  letters  (e.g., 
Social  Influence  [SI]).  A  brief  description  of  the  module's 
contents  was  also  provided.  The  list  was  intended  to  serve  as 
an  easy  reference  tool  by  interviewers  to  help  them  become 
familiar  with  the  various  sections  of  the  survey  instrument 
and  their  purposes.  The  list  was  included  in  the  Training 
Manual  and  was  also  provided  on  a  separate  page  for  easy 
reference . 

(2)  Respondents'  Question  List  (Figure  4).  A  two-page  list  of 
anticipated  respondent  questions  was  also  provided  along  with 
suggested  answers  for  each.  It  was  included  in  the  Training 
Manual  as  well  as  on  separate  cards  for  easy  reference  while 
interviewing. 

(3)  Magazine  List  ^Figure  51 .  A  magazine  list  was  provided  for 
use  in  the  Media  Habits  module,  where  respondents  were  asked 
to  name  the  magazines  they  read  on  a  regular  basis.  The 
magazine  list  provided  an  extensive  alphabetical  enumeration 
of  popular  magazines  paired  with  three-digit  code  numbers  for 
interviewers'  use  in  recording  respondents'  answers. 
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Household  Scmur  (SC):  This  MCtioa  has  questions  designed  to  locate  households  with 
members  eligible  for  inclusion  in  ooo  of  tho  uaplos.  Bnsod  oo  responses  a  th«$o  questions 
th«  computer  will  docids  if  U  appropriate  respondent  has  baoa  found  or  if  tha  interview 
should  ba  taraiaatad. 

Educatloa  and  Employment  (EE):  Youth  will  ba  askad  quastioos  rtgardini  thair  aducatioaal 
and  work  experiences. 

Caraar  Military  lataatiaas  or  Prepeaslty  (IP):  This  sactioo  aaasuras  tha  youth's  pareaivad 
likaiihood  that  ha  or  sha  will  aalist  ia  tha  Amy  or  will  chooaa  soma  othar  caraar  opuoa  id 
military  or  civilian  Ufa. 

Behavior*  (BE):  This  taction  includes  quastioos  oa  tho  youth's  axpioratioa  of.  or  iatarast  ia. 
various  caraar  options,  including  tha  Amy. 

Social  taflaeace  (SI):  Tha  influaoca  of  family,  friends  and  educator*  has  been  considered 
important  ia  tha  decisioa  to  aalist.  This  saetioa  contains  questions  about  what  the  youth 
perceives  to  ba  tha  altitudes  of  key  parsons  towards  his  or  her  joining  tha  Amy. 

Importance  of  Kay  Attribute*  (IA):  This  section  contains  questions  oa  tha  importance  of 
various  factors,  such  u  giving  service  to  your  country  and  developing  one's  potential,  that 
may  influaoca  caraar  choices. 

Media  Habits  (MH):  Questions  her*  fall  iato  two  classes:  (1)  items  focused  on  the 
respondent's  TV  viewing,  radio  listening,  and  magazine  and  newspaper  reeding  habits,  and 
(2)  items  focused  oa  specific  shows  and  magazines  that  might  have  been  seen,  heard,  or 
read. 

Knowledge  Recall  (KB):  A  basic  way  to  measure  how  knowledgeable  respondents  are  about 
military  communications  is  to  ask  whether  they  recall  Army  and  other  services' 
advertisements  ia  general,  and  particular  message*  in  those  advertisements  specifically. 

Attitude*  towards  Amy  Advertising  (AT):  This  section  measures  how  much  respondents 
like  Army  ads  and  how  much  they  baliav*  what  the  ads  say. 

Kaowledge/Slogaa  Recogeldaa  (KSh  Respondents  will  ba  asked  to  match  slogans  in 
military  advertisements  with  tha  services  that  us*  them. 

Perceptieas/Bellefs  of  Amy  Attribute*  (PE):  (urns  in  this  section  ask  whether  serving  in 
the  Army  offer*  certain  opportunities,  such  u  developing  leadership  skills.  The  second  set 
of  items  focus  oa  perceptions  about  people  who  join  tho  Army. 

Kaewledge/Awareaeee  (KA):  These  items  focus  on  specific  Army  incentives,  such  as  pay, 
education  benefits,  and  -training. 

Parental  Influence  (PI):  This  section  asks  parents  about  actions  they  have  taken  to 
influence  thair  child's  career  dee  is  mo-making  process. 

Demographics  (DE):  Items  here  uk  about  parents  about  actions  they  have  taken  to 
influence  their  child's  career  dec  is  ton-making  process. 

Parental  Location  (PL):  A  randomly  selected  portion  of  youth  will  be  asked  (or  the  name 
and  telephone  number  of  one  of  their  parents,  who  will  then  be  interviewed. 

Tracking  information  (TR):  The  youth  chosen  to  b*  pan  of  a  longitudinal  subtample  will 
be  asked  for  information  which  will  make  it  tasiar  to  recontact  them  for  the  follow. up 
interview. 


Fiaure  3.  Questionnaire  modules. 
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•I  hart  time.* 

We  cu  do  it  at  aay  tuna  that  is  eoavtaicat  for  you.  (Pause)  If  you  don't  have  time  now.  just 
tail  me  when  aad  1  caa  call  you  back.  Your  participation  is  that  important  to  the  success  of  the 
survey. 

*How  long  will  this  Interview  take?* 

//  tht  qutstton  rt/trt  to  tht  tcrttntr: 

The  first  part  of  this  survey  is  very  short.  It  will  take  about  3  minutes  aad  that  may  be  all  the 
time  we  need.  If  we  need  more  time  for  additional  questions,  I  will  let  you  know  at  that  point. 

If  tht  outslio*  rtftrs  to  tht  Youth  irutnitw. 

Depending  on  your  answers,  it  will  take  between  20  and  34  minutes. 

If  tht  fut st  ion  rtftrs  to  tht  fartnt  iiutrtitw. 

Depending  on  your  answers,  it  will  take  about  13  or  20  minutes. 

*How  did  yea  get  my  telephone  asmbcrT* 

Your  number  was  randomly  dialed  from  a  list  of  all  telephone  prefixes  used  in  the  United 
States. 

•Why  can't  yon  Interview  someoae  else?* 

We  can't  substitute  someone  else.  If  we  did  that,  the  sample  would  not  bo  scientifically  accurate 
and  we  couldn't  roly  on  the  results  we  got. 

‘Why  do  yon  want  to  talk  with  me?* 

We  are  surveying  only  young  adults  in  a  particular  age  range,  and  sometimes  their  parents  If 
there  are  other  persons  in  your  household  in  this  age  range,  you  were  the  one  who  was 
randomly  chosen  to  be  included  in  the  survey. 

•How  do  I  know  yon  are  who  yon  aay  yon  are?* 

If  you  wish,  you  caa  call  my  supervisor  .  She/he  caa  be  reached  at  100-423-6754  and  can  give 
you  all  the  particulars  of  the  study. 

’Do  I  have  to  answer  your  question*?* 

No,  your  participation  in  the  study  is  entirely  voluntary,  nod  you  will  not  be  urged  to  answer 
aay  question  that  makes  you  uncomfortable.  Since  your  name  will  never  be  released  in 
connection  with  the  survey,  though,  we  hope  that  you  will  choose  to  participate. 

*How  do  I  know  yon  will  keep  thU  laferssatioa  coafldeadal?* 

We  are  required  by  law  not  to  reveal  any  information  exeept  to  the  Wastat  research  staff 
working  »n  the  study.  Each  of  us  is  required  to  sign  a  statement  to  keep  as  confidential  all 
information  provided  by  respondents.  Survey  results  will  be  published  only  as  statistical  totals. 
No  information  which  would  permit  the  identification  of  an  individual  will  be  released  or 
published. 

*What  la  the  background  noise  I  am  hearing?* 

Your  answers  are  being  entered  directly  into  our  computer  system. 


Figure  4. 


Respondent' s 


question  list. 
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■What  U  Ik#  purpose  of  (bo  sareey?* 

The  survey  focusos  on  the  career  pleasing  of  young  adults  and  how  iaformatioa  about  miliary 
service  figures  ia  their  thinking.  Th«r«  ere  also  soma  questions  on  tha  messages  people  receive 
from  advertisements  sponsored  by  the  military. 

•Who  Is  spoaserieg/faadlag  this  study?* 

This  study  is  beiag  conducted  for  the  U.S.  Army  Research  Institute  ia  Washington,  D.C. 

•What  types  of  qoeetlooe  will  yoa  ask  me?* 

The  interview  will  generally  be  about  what  you  have  seen,  heard,  or  read  about  serving  ia  the 
Army  today.  We  will  also  ask  about  your  plans  for  the  future. 

•will  the  Army  kaow  yea  talked  with  me  aad  what  I  said?* 

We  won't  bo  asking  the  name  of  most  of  the  people  we  will  survey.  Information  that  might 
identify  specific  persons,  such  as  names,  addressee  and  telephone  numbers  will  never  be  given 
to  the  Army. 

•Do  yoa  work  for  the  Army?*  or  ’Are  yoa  a  Recruiter?* 

No,  I  am  employed  by  Westat  Incorporated,  a  survey  research  firm  located  ia  the  Washington, 
D.C.  metropolitan  area.  We  are  conducting  this  survey  under  contract  with  the  U.S.  Army 
Research  Institute. 

*1  had  a  bad  experience  with  the  goverameat/mllltary.*  or  *r«e  never  liked  the  military.  Why 
should  I  help  them  by  aasweriag  yoar  questions?* 

This  is  a  special  research  effort,  aad  we  are  interested  in  your  opinions  whether  they  ere 
positive  or  negative. 

•Cna  yoa/will  yoa  give  my  aeme  te  a  recruiter?* 

No.  we  ere  only  conducting  e  survey  and  htve  no  contacts  with  Army/military  recruiters. 

•How  will  the  survey  resell*  be  used?* 

The  Army  needs  to  know  whtther  young  adults  end  their  parents  ere  picking  up  the  messages 
Army  advertising  is  trying  to  send.  If  the  messages  are  not  being  received,  the  advertising  has 
to  chtngt. 

The  messages  concern  what  it's  like  to  be  in  the  Army.  The  Army  is  an  all-volunteer  force 
that  relies  upon  young  adults  viewing  it  favorably  end  enlisting  for  duty.  If  the  Army  can 
affectively  communicate  what  it  offers,  young  adults  can  make  more  informed  career  decisions 
regarding  miliary  service. 

•Why  do  you  went  my  Social  Security  Number?* 

Providing  us  with  your  Social  Security  number  is  voluntary.  We  went  to  tae  whether  any  of  the 
ideas  we've  been  discussing  are  related  to  whether  or  not  someone  eventually  enlists  in  the 
Army.  No  one  ia  the  Army  or  anyone  else  in  the  government  will  ever  see  your  Social  Security 
Number  or  anything  alio  that  could  be  used  to  identify  you. 

Only  research  project  staff  would  have  access  to  your  Social  Security  Number.  Each  member  of 
the  research  staff  ia  required  to  sign  a  statement  never  to  reveal  confidential  information. 

•Will  you  ese  my  Social  Security  aemher  to  leek  el  ether  geveremeat  records  ea  me?* 

Only  the  records  of  the  Army  will  be  consulted  by  Wesmt  staff.  We  simply  went  to  see  whether 
people  with  certain  attitudee  towvd  the  Army  eventually  enlist  in  it.  We  an  required  by  taw 
not  to  reveal  any  information  ascept  to  tha  Westat  research  staff  working  on  the  study.  Each  of 
us  is  required  to  sign  t  statement  to  keep  is  confidential  ell  informntion  provided  by 
retpoadeaa. 


Figure  4.  Respondent's  question  list,  (continued) 
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Data  Collection  Operations 


Interviewing  Periods 

Each  monthly  sample  of  telephone  numbers  was  active  for  an  eight 
week  period,  beginning  on  the  first  day  of  the  month,  during  this 
period,  attempts  were  made  to  contact  all  telephone  numbers  in  the 
sample,  to  separate  households  from  non-households,  to  conduct  the 
household  screening  interviews,  and  to  complete  the  youth  interviews. 

Attempts  were  made  to  initiate  the  parental  interviews  as  soon  as 
each  parental -linked  youth  interviews  were  completed.  The  period  for 
interviewing  the  parental  sample  extended  two  weeks  beyond  the  close  of 
the  eight  week  period  designated  for  the  youth  interviews. 

This  schedule  implied  that,  at  any  one  point  in  time,  several 
sample  cohorts  were  available  for  interviews.  A  complex  data  collection 
management  system  was  therefore  necessary  to  monitor  data  collection 
progress  for  each  monthly  cohort. 

Staffing 

The  Frederick  Telephone  Research  Center  (TRC)  operated  from  9  a.m. 
to  12  p.m.  on  Mondays  through  Fridays,  10  a.m.  to  6  p.m.  on  Saturdays, 
and  2  p.m.  to  10  p.m.  on  Sundays.  Interviewers  were  scheduled  on  the 
ACOMS  project  during  all  time  periods  to  permit  adequate  coverage  of 
households  located  in  the  four  time  zones  of  the  United  States.  Inter¬ 
viewer  shifts  ranged  from  four  to  eight  hours. 

Weekly  project  meetings  held  at  the  TRC  facility  included  the 
facility  manager,  ACOMS  project  director  and  other  project  staff,  TRC 
project  supervisor,  key  ACOMS  CATI  programmers,  and  the  data  preparation 
supervisor.  Topics  discussed  during  these  weekly  meetings  included 
effective  interviewer  scheduling,  training  needs,  work  flow,  CATI  hard¬ 
ware  and  software,  data  cleaning  and  coding,  and  project  procedures. 

The  TRC  operations  manager  functioned  as  the  primary  liaison 
between  the  TRC  and  ACOMS  project  staff.  The  manager  worked  closely 
with  the  project  supervisor  and  shift  supervisors.  The  operations 
manager  and  project  supervisor  reviewed  all  problem  cases  and  were 
responsible  for  establishing  procedures  for  the  resolution  of  these 
cases . 

Quality  Control 

Project  supervisors  were  located  "on  the  floor"  with  the  inter¬ 
viewers  during  all  hours  of  TRC  operation.  Using  extension  telephones 
and  displays  linked  to  the  interviewers'  CRTs,  supervisors  monitored 
approximately  10X  of  each  interviewer's  work  during  the  study.  Each 
time  an  interview  was  monitored,  the  supervisor  would  complete  a 
monitoring  form.  At  the  completion  of  the  interview,  the  supervisor 
provided  feedback  to  the  interviewer  regarding  his/her  techniques  of 
gaining  cooperation  and  asking  questions,  and  the  accuracy  of  responses 
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recorded.  Problems  commonly  encountered  by  interviewers  were  discussed 
in  memos  and  regular  interviewer  meetings. 

Supervisors  were  responsible  for  monitoring  interviews  and  ensuring 
that  problems  were  quickly  identified  and  resolved.  Whenever  it  was  not 
possible  to  resolve  a  problem  immediately,  a  CATI  Problem  Sheet  was 
completed  by  the  interviewer  and  brought  to  the  attention  of  the  TRC 
project  supervisor  for  review  and  resolution.  This  sheet  provided  space 
for  the  interviewer  to  record  the  pertinent  identifying  information 
regarding  the  case  and  a  complete  description  of  the  problem  encoun¬ 
tered.  A  CATI  Update/Comment  Sheet  was  also  completed  by  interviewers 
whenever  they  needed  to  have  a  data  item  corrected,  e.g. ,  a  screener 
respondent  provided  an  incorrect  spelling  of  a  youth's  name  which  the 
youth  subsequently  corrected.  The  interviewers  used  this  sheet  to 
record  the  case  identifying  information  and  the  corrected  information. 
After  reviewing  the  information  contained  on  these  forms  and  the 
information  recorded  on  the  CATI  file,  the  project  supervisor  would 
route  the  sheets  to  the  data  preparation  staff  if  changes  in  the  data 
file  were  necessary. 

A  core  of  experienced  interviewers  was  identified  and  trained  to 
assist  in  resolution  of  problem  procedure  cases,  e.g.,  during  subsequent 
contacts  with  a  household  there  was  a  discrepancy  in  the  description  of 
household's  composition.  All  problem  cases  were  reviewed  thoroughly  by 
the  project  supervisor  and  operations  manager  and  subsequently  routed  to 
either  the  CATI  programmers  or  data  cleaning  staff  as  appropriate. 

Data  quality  was  also  ensured  by  regular  reviews  of  data  frequen¬ 
cies.  These  reviews  ensured  that  correct  branching  occurred  and  the 
data  were  being  stored  properly.  Irregularities  were  discussed  and 
resulted  in  data  changes  or  minor  revision  in  CATI  programming. 

Production  Reporting  and  Management  Systems 

In  order  to  facilitate  management  of  the  complex  data  collection 
effort,  numerous  reports  were  developed.  These  reports  contained 
summaries  of  pertinent  information  at  both  the  interviewer  and  project 
level . 

Interviewer  Reports 

The  Telephone  Information  Management  System  (TIMS)  report  provided 
weekly  information  regarding  the  total  number  of  hours  scheduled  on  the 
project  by  day  of  week  and  by  day/evening/weekend  hours.  This  informa¬ 
tion  was  available  on  a  projected  basis  for  the  upcoming  week  and  as 
actual  hours  worked  for  the  previous  week.  A  cumulative  summary  of  this 
information  was  provided  for  the  project  staff.  Additional  reports 
listing  projected  and  actual  hours  worked  for  individual  interviewers 
were  also  generated  on  a  weekly  basis. 

Another  report  displayed  the  number  of  people  logged  onto  the  ACOMS 
account  every  hour  the  project  was  operational  during  a  given  week. 

This  report  was  useful  in  comparing  the  number  of  interviewers  projected 
to  be  working  at  a  given  time  with  the  actual  number. 
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Weekly  interviewer  reports  reflected  the  final  disposition  of  each 
finalized  case  and  response  rates  on  an  individual  interviewer  level. 
These  interviewer-specific  reports  enabled  the  supervisory  staff  to 
identify  interviewers  with  a  low  level  of  productivity  or  staff  who 
required  additional  training. 

Interviewing  Status  Reports 

Various  project- level  reports  were  also  generated  to  track  the  work 
flow  and  response  rates  for  the  different  components  of  the  study,  i.e., 
screener,  youth,  and  parent  interviews.  These  reports  reflected  the 
status  of  all  sample  telephone  numbers  by  final  disposition,  such  as 
complete,  final  refusal,  out-of-scope ,  and  interim  dispositions  such  as 
callbacks,  first  refusal,  etc.  The  Project  Report  tracked  response 
rates  for  each  interview  component  before  and  after  refusal  conversion 
efforts.  A  unique  feature  of  the  Daily  Report  for  ACOMS  study  was  a 
calendar  displaying  the  daily  counts  by  sample  month  of  completed  out- 
of-scope  numbers,  i.e.,  non-residential  (NR)  and  non-working  (NW) ,  and 
completed  screener,  youth  and  parental  interviews. 
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5.  ACOMS  SURVEY  SAMPLE  WEIGHTING,  FACTORS  AFFECTING  SAMPLE 
PRECISION,  AND  ANALYSIS  METHODOLOGY 


Michael  J  Wilson  and  Leyla  Mohadjer 


Introduction 


This  chapter  provides  a  discussion  of:  (a)  the  methodology  used  for  the 
calculation  of  survey  sample  adjustment  weights,  (b)  factors  affecting  the 
precision  of  sample  estimates,  and  (c)  the  methods  used  for  testing  the 
statistical  significance  of  parameter  estimates  utilized  in  the  analysis  of 
data  from  the  Army  Communications  Objectives  Measurement  System  (ACOMS) 
survey . 

The  ACOMS  youth  and  parent  samples  were  drawn  utilizing  clustering, 
stratification,  and  supplementation  techniques  in  order  to  assure  compliance 
with  subpopulation  sample  size  requirements.  As  a  consequence,  respondents 
were  selected  with  differing  probabilities  of  sample  selection  depending  upon 
demographic  and  other  characteristics.  These  departures  from  a  simple  random 
sample  (SRS)  design  complicated  the  processing  and  analysis  of  ACOMS  survey 
data. 

Sample  adjustment  weights  had  to  be  calculated  and  used  in  order  to 
produce  unbiased  estimates  of  ACOMS  population  parameters.  For  example, 
without  the  use  of  adjustment  weights,  the  ACOMS  youth  sample  would  estimate 
the  proportion  of  males  and  females  in  the  population  as  .83  and  .17, 
respectively.  Sample  adjustment  weights  are  used  to  compensate  for  unequal 
selection  probabilities  among  the  ACOMS  subpopulations.  This  chapter  presents 
the  methodology  followed  in  weighting  the  ACOMS  youth  sample  and  calculating 
adjustment  weights  for  the  perceptions  and  noncore  questionnaire  modules  as 
well  as  for  the  parental  sample. 

Survey  sample  design  and  design  related  factors  such  as  sample  weighting 
adjustments  affected  the  precision  of  sample  estimates.  For  example, 
departures  from  a  SRS  design  such  as  the  cluster  sampling  used  in  ACOMS 
increased  the  variance  of  population  parameter  estimates  relative  to  that 
which  would  be  expected  from  a  SRS  design.  In  general,  while  complex  sample 
designs  can  markedly  improve  the  effectiveness  of  data  collection,  they  do  so 
at  the  expense  of  efficiency  in  variance  estimation.  The  term  used  to 
describe  and  quantify  this  relative  (to  a  SRS  design)  efficiency  is  design 
effect.  In  addition,  the  use  of  sample  adjustment  weights  can  affect  the 
precision  of  estimates.  If  sample  adjustment  weights  are  highly  variable 
(i.e.,  unequal),  the  variance  of  estimates  will  be  greater  than  would  be 
expected  if  all  weights  were  equal.  The  term  used  to  describe  this  source  of 
increase  in  variance  is  unequal  weighting  effect.  This  chapter  discusses  the 
calculation  and  magnitude  of  ACOMS  design  and  unequal  weighting  effects  for 
various  sampled  subpopulations. 

The  use  of  a  complex  sample  design  also  had  implications  for  the 
statistical  analysis  conducted  on  ACOMS  data.  In  order  to  perform  tests  of 
statistical  significance,  two  methodologies  were  utilized.  For  contingency 


table  analysis  an  approximate  chi-square  statistic  was  computed  which  took 
into  account  actual  sample  size  and  the  magnitude  of  unequal  weighting 
effects.  All  other  significance  tests  were  accomplished  using  the  method  of 
balanced  repeated  replications  (BRR) .  This  chapter  ends  with  a  brief 
description  of  the  approximate  chi-square  statistic  and  BRR  methodology. 


Weighting  the  ACOMS  Sample  Data 


The  ACOMS  sample  design  did  not  produce  a  self-weighted  sample  of 
individuals  who  are  members  of  the  eligible  youth  population.  Sample 
adjustment  weights,  with  different  weights  for  various  subdomains  of  the 
population  that  have  been  sampled,  were  necessary  for  the  production  of 
unbiased  parameter  estimates.  (In  addition,  a  second  set  of  weights  were 
produced  for  the  Perceptions  and  non-core  questionnaire  modules).  Sample 
adjustment  weights  were  used  with  ACOMS  data  to  provide  estimates  of 
statistics  (means,  proportions,  correlations,  etc.)  that  would  have  been 
obtained  if  the  entire  sampling  frame  had  been  surveyed.  The  ACOMS  weighting 
methodology  was  implemented  to  accomplish  the  following  objectives: 

(1)  to  bring  sample  data  up  to  the  dimensions  of  population 
totals ; 

(2)  to  adjust  for  differential  probabilities  of  selection  among 
subgroups  (Hispanics,  females)  of  the  population; 

(3)  to  minimize  biases  arising  from  the  fact  that  nonrespondents 
may  differ  from  survey  respondents; 

(4)  to  compensate  for  inadequacies  in  the  sampling  frame  (the 
sampling  frame  excluded  nontelephone  households  and  possibly 
persons  living  in  unconventional  settings) ;  and 

(5)  to  reduce  variances  of  estimates  by  using  auxiliary 
information  that  is  known  with  a  high  degree  of  accuracy  in 
the  estimation  procedure. 


Three  distinct  sets  of  weighting  adjustments  were  calculated  for  the 
ACOMS  data  depending  upon  the  sample  and  questionnaire  modules  under 
consideration.  First,  youth  sample  weights  were  calculated  for  the  entire 
sample.  Next,  adjustments  were  calculated  for  perceptions  and  non-core 
questionnaire  modules.  Finally,  adjustment  weights  were  computed  for  the 
ACOMS  parental  sample. 

Youth  Sample  Weights 

Calculation  of  sample  adjustment  weights  for  the  ACOMS  youth  sample  was 
accomplished  in  three  steps.  The  first  two  steps  involved  computation  of 
weights  compensating  for  unequal  probabilities  of  selection  at  the  household 
and  person  level.  The  third  step  used  poststratification  (also  referred  to  as 
ratio-estimation)  to  adjust  for  sample  nonresponse,  compensate  for  the 
omission  of  nontelephone  households,  and  reduce  sampling  errors. 
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Sampling  rate  adjustments  at  the  household  level.  The  modified  Waksberg 
method  was  used  to  sample  households.  In  this  approach,  a  constant  number  of 
telephone  numbers  per  cluster  (rather  than  households,  as  in  the  standard 
method)  were  selected.  As  a  result,  households  had  differential  probabilities 
of  selection.  The  rate  at  which  a  household  was  sampled  depended  on  the 
proportion  of  telephone  numbers  within  a  cluster  proving  to  belong  to 
households.  As  a  result,  a  weight  was  attached  to  each  cluster  equal  to  the 
average  number  of  sampled  households  per  cluster  divided  by  the  number  found 
in  the  particular  cluster.  That  is, 


Wi 


where  n  is  the  average  number  of  households  per  cluster,  and  n^  is  the  actual 
number  of  sample  households  in  the  iC^  cluster. 


Additionally,  households  with  two  telephone  numbers  had  twice  the  chance 
of  selection  and  were  over  represented  in  the  sample .  Thus ,  they  were  given  a 
weight  of  1/2  to  adjust  for  this  overrepresentation.  Variable  I—  was  defined 
in  the  following  way: 

Ijj  -1  if  household  j  in  the  ic^  cluster  had 
one  telephone  number,  or 

-  1/2  if  household  j  in  iC^  cluster  had  more 
than  one  telephone  number. 

The  household  level  or  first  stage  weight,  then,  was  defined  as 
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I  1/2 

1  or 

I  1.0 


(1) 


It  was  observed  during  the  weighting  of  data  from  the  first  quarter  of 
data  collection  that  a  small  proportion  (approximately  3  percent)  of  clusters 
had  fewer  than  four  screened  households.  As  a  result,  respondents  from  these 
clusters  were  assigned  very  large  household  weights  which  correspondingly 
(  increased  sampling  variances  (due  to  considerable  variability  in  the 

magnitudes  of  household  weights) .  To  avoid  this  problem  of  disproportionately 
large  household  weights  (and  sampling  variances)  in  the  second  and  subsequent 
quarters,  constraints  were  placed  on  the  magnitude  of  acceptable  household 
weights.  Specifically,  weights  assigned  to  clusters  with  fewer  than  four 
screened  households  were  set  equal  to  the  weight  for  clusters  with  four 
|  screened  households. 
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This  approach  produced  smaller  sampling  variances  at  the  price  of 
introducing  some  unknown  bias  due  to  under-representation  of  households 
obtained  from  clusters  having  one,  two,  or  three  screened  households.  (The 
bias  depends  on  unknown  factors  such  as  nonresponse.)  Since  a  very  small 
proportion  of  clusters  had  fewer  than  four  screened  households,  it  is  assumed 
that  the  bias  is  negligible  in  comparison  to  the  gains  obtained  from  reducing 
the  sampling  variances . 

Sampling  rate  adjustments  at  the  person  level.  The  ACOMS  survey  design 
dictated  the  sampling  of  various  population  subgroups  at  different  rates.  For 
example,  Hispanics  were  oversampled  and  females  undersampled.  It  was 
necessary,  therefore,  that  sample  weights  adjust  for  the  differential  sampling 
rates  used  for  population  subgroups  in  addition  to  adjustments  for  sample 
selection  probability  at  the  household  level.  Sample  rate  weighting 
adjustments  calculated  for  each  person  reflected  his  or  her  actual  probabili  ty 
of  selection.  This  adjustment  was  made  by  multiplying  each  person's  first 
stage  (i.e.,  household)  weight  by  the  reciprocal  of  the  probability  of 
selection  for  the  subdomain.  The  second  stage  or  person-level  weight  was 
defined  as 


u2ijk  -  “lij  '  “ijk  <2> 

where  is  the  reciprocal  of  the  probability  of  selection  for 

the  kth  individual  in  the  j household  in  cluster  i. 

These  adjustments  ensure  that  the  weighted  counts  or  proportions  for 
females  and  Hispanics,  for  example,  reflected  their  composition  in  the  youth 
population  rather  than  the  sample  design.  This  was  very  important  as 
unweighted  counts  or  proportions  would  be  very  misleading  when  used  as 
estimates  of  population  values. 


Poststratification.  Poststratification  was  used  to  reduce  sampling 
errors,  to  minimize  biases  arising  from  differences  between  respondent  and 
nonrespondent  groups,  and  adjust  for  the  exclusion  of  nontelephone  households 
from  the  sampling  frame.  Poststratification  adjusted  the  weights  calculated 
in  the  first  two  stages  of  weighting  by  creating  agreement  between  weighted 
ACOMS  tabulations  and  census  estimates  of  the  total  population  by  age,  sex, 
race  and  ethnicity. 

Nonresponse  and  inadequacies  in  the  ACOMS  sampling  frames  were  expected 
to  vary  by  population  groups  and  therefore  potentially  distort  sample 
distributions.  Poststratification  compares  the  distribution  of  the  population 
and  the  sample  across  selected  variables  and  computes  sample  weights  to  ensure 
that  sample  proportions  on  key  demographic  characteristics  closely 
approximates  known  population  proportions.  When  post-stratification  is  not 
applied  to  sample  data,  obtained  weighted  sample  distributions  may  differ 
substantially  from  those  in  the  population. 


There  was  a  difference  in  the  poststrata  formed  for  the  weighting  of  the 
first  three  quarters  of  data  collected  and  those  used  in  weighting  year  two 
data.  This  difference  will  be  documented  following  a  discussion  of  the  post¬ 
stratification  methodology  as  applied  to  the  first  three  quarters  of  ACOMS 
data. 

Four  variables  were  used  for  construction  of  poststrata  for  the  first 
three  quarters  of  data: 


(1) 

Brigade : 

5  categories 

(2) 

Sex: 

2  categories 

(3) 

Race/Ethnicity : 

3  categories 

(Black  non- 

Hispanic/ 

Hispanic/Other) 

(4) 

Age: 

males  4  categories 

(16-17/18-19/ 
20-21/22-24 
years  old)  and 


females  2  categories 

(16-19/20-24 
years  old) 

Poststratification  by  these  variables  was  necessary  to  assure  that  sampling 
proportions  on  these  key  variables  were  similar  to  population  proportions  and 
to  allow  Army  analysts  to  make  comparisons  of  ACOMS  findings  with  other  survey 
efforts  such  as  the  Youth  Attitude  Tracking  Survey  (YATS) . 

This  strategy  yielded  the  following  poststrata  cells  for  the  first  three 
quarters  of  ACOMS  data. 


Males 

ROWS 

by  race/ethnicity 
by  age 

COLUMNS 

Brigade 

Females 

by  race/ethnicity 
by  age 

There  is  a  practical  problem  in  calculating  poststratification  weights 
when  the  number  of  crosstabulation  cells  is  very  large.  Some  of  the  cells  may 
contain  no  sampled  individuals  while  others  may  contain  only  a  small  number. 

To  avoid  problems  arising  from  small  sample  sizes  within  poststrata  cells, 
raking  (also  termed  iterative  proportional  fitting)  was  used  to  compute  the 
adjustment  weights  for  the  poststrata.  Raking  is  an  iterative  procedure  that 
estimates  weights  so  that  weighted  sample  estimates  equal  population  controls 
for  the  marginals  of  the  various  poststrata  without  ensuring  this  quality  for 
each  of  the  crosstabulation  cells.  That  is,  raking  will  ensure  that  weighted 
marginal  distributions  of  recruiting  brigades,  sex,  race/ethnicity,  and  age 
will  agree  with  known  population  marginals  but  raking  does  not  attempt  to 
achieve  a  correspondence  between  sample  and  population  values  in  the  cells  of 
the  crosstabulation. 


The  poststratification  weights  W^lnm  were  computed  so  that 


E 

lmn 

E 

hmn 

E 

hln 

E 

him 


Whlmn  W2ijk  "  N 


Whlmn  W2ijk 


h.  .  . 


Whlmn  W2ijk  -  N 


.1.  . 


Whlmn  W2ijk  “  N..m.’ 


.  .n’ 


h  -  1,2, 4, 5, 6 

-  brigade; 

1  -  1,2 

-  sex; 

m  -  1,2,3 

-  race/ethnicity;  and 
1,2, 3, U  if  1-1 


(3) 

(4) 

(5) 

(6) 


-  age. 


To  simultaneously  solve  equations  (3)  to  (6)  for  the  poststratification 
weights,  the  raking  procedure  proceeds  iteratively  by  proportionately 
weighting  poststrata  cell  values  so  that  each  of  the  equations  is  satisfied  in 
turn.  Each  new  iteration  begins  from  the  results  of  the  previous  step  and 
continues  until  all  equations  are  satisfied  to  the  degree  of  precision 
required. 

The  weight  for  each  individual  during  the  first  three  quarters  of  data 
collection  was  the  product  of  the  poststratification  weight  and  W2iik' 
computed  in  equation  (2). 


^Ihijklmn  ~  ^hlmn  *  ^2ijk 


(7) 


In  the  second  year  of  data  collection,  education  was  added  as  a  post¬ 
stratification  variaole.  The  poststrata  cells  for  the  second  year  of  data 
collection  were : 
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Males 

ROWS 

by  race/ethnicity 
by  age 

COLUMNS 

Brigade  by  Education 
of  head  of  household 

Females 

by  race/ethnicity 
by  age 

The  addition  of  another  dimension  to  poststratification  added  one  more 
subscript  to  the  weight  and  another  equation  to  satisfy  during  raking.  The 
methodology  remained  unchanged,  however.  The  weight  for  the  second  year  of 
data  was 


W3hijklmno  *  Whlmno  *  W2ijk' 

The  March  supplement  of  the  1986  Current  Population  Survey  (CPS)  was 
used  for  the  poststrata  population  counts  for  the  first  three  quarters  of  data  • 

collection  and  the  March  supplement  of  the  1987  CPS  was  used  for  the  second 
year.  Poststrata  population  counts  were  based  on  the  CPS  estimates  for  16-  to 
24-year-old  youths  excluding  in-service  youth  and  college  graduates.  The  CPS 
estimate  of  in-service  youth  did  not,  however,  include  those  who  had  been 
accepted  into  a  branch  of  the  military  and  were  waiting  to  go  on  active  duty 
(a  characteristic  of  the  ACOMS  sample  definition)  or  a  reliable  estimate  of  .• 

the  number  of  youth  with  prior  military  service.  To  estimate  population 
proportions  for  these  groups,  information  available  from  the  ACOMS  screener 
regarding  youth  waiting  to  enter  the  military  and  those  with  prior  service  was 
used.  The  poststrata  population  counts  obtained  from  the  CPS  were  then 
adjusted  to  exclude  these  groups.  These  two  estimated  proportions  remained 
stable  throughout  the  data  collection  period.  JJj 

Final  poststratification  weighting  adjustments  were  trimmed  in  a  manner  I 

analogous  to  that  used  in  the  establishment  of  household  level  weights.  The  j 

purpose  served  was  the  same  as  well  (i.e.,  to  reduce  variance  inflation  due  to  j 

highly  variable  adjustment  weight  values).  Sample  poststratification  weights  j 

were  first  examined  for  unacceptably  large  values  and  the  cells  containing  # 

these  weights  identified.  The  largest  weight  was  then  constrained  to  be  equal 
to  the  value  of  the  next  highest  weight.  The  difference  between  these  weights 
was  then  distributed  proportionately  among  the  remaining  weights  in  the  j 

poststratum . 

• 
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Adjustment  Weights  for  the  Perceptions  and  Non-Core  Modules 


As  described  more  fully  in  Chapter  3  (Allen,  Nieva,  &  Gaertner,  1988), 
sections  of  the  Perceptions  module,  the  rotating  modules  (Media  Habits,  Slogan 
Recognition  and  Knowledge  of  Army  Offers)  and  the  Social  Influences  and 
Parental  Location  modules  were  administered  to  designated  subsamples  of 
respondents.  The  ACOMS  Perceptions  module  included  questions  referring  to 
perceptions  of  the  active  Army,  ROTC,  National  Guard,  and  U.S.  Army  Reserve, 
perceptions  of  other  military  services  (Air  Force,  Navy,  and  Marines),  and 
perceptions  of  other,  nonmilitary  options  (i.e.,  attending  college  and  in 
civilian  employment)  open  to  respondents  in  targeted  age  groups.  In  order  to 
restrict  respondent  burden  to  an  average  of  30  minutes  per  questionnaire 
administration,  respondents  were  only  asked  their  perceptions  about  two,  or  at 
most  three,  of  the  referents  identified  above.  The  distribution  of 
perceptions  questions  by  respondents  varied  across  the  military  and  civilian 
option  referents  depending  on  the  allocation  formula  described  in  Chapter  3 
(Allen,  Nieva,  &  Gaertner,  1988)  of  this  document.  For  example,  almost  all 
college  students  were  asked  their  perceptions  of  ROTC,  whereas  only  a  small 
portion  of  college  students  were  asked  their  perceptions  of  the  Navy  or 
Marines . 

Similarly,  random  half-samples  of  respondents  were  administered  the 
Media  Habits,  Slogan  Recognition,  Knowledge  of  Army  Offers  questionnaire 
modules.  (These  are  termed  "rotating  modules.")  Additional  sets  of  weights 
were  computed  reflecting  the  sampling  rates  used  for  selecting  respondents  for 
these  modules  and  perception  sections.  The  weights  were  equal  to  the  product 
of  the  reciprocal  of  selection  probability  for  the  rotating  modules  or 
perceptions  sections  and  the  sampling  weight  computed  in  equation  (7)  or  (8) 
as  appropriate. 


The  procedures  for  weighting  the  Social  Influence  and  Parent  Location 
modules  were  slightly  different.  These  modules  were  only  asked  of  those  16- 
to  20-year-old  Primary  Male  Sample  (PMS)  and  Primary  Female  Sample  (PFS)  youth 
who  were  selected  for  parental  interviews.  Only  one  age-eligible  PMS/PFS 
youth  per  household  was  selected  to  be  part  of  the  parental  linked  sample. 
Therefore,  the  youth  base  weights,  as  computed  in  equation  (2),  were  adjusted 
for  multiple  number  of  eligibles  in  households.  Furthermore,  the 
poststratification  counts  had  to  be  adjusted  to  include  only  the  16-  to  20- 
year-old  PMS/PFS  youth  in  the  population.  Since  no  reliable  estimate  of 
Primary  and  Secondary  populations  was  available,  the  proportion  obtained  in 
the  samples  was  used  as  an  estimate  of  the  population  proportion  and  the  CPS 
counts  were  adjusted  accordingly. 


Parent  Sample  Weights 


The  computation  of  parent  sample  adjustment  weights  followed  the 
computation  of  the  youth's  weights  for  the  Social  Influence  and  Parent 
Location  modules.  The  parent  base  weight  was  initially  set  equal  to  the 
designated  youth  weight  for  these  modules.  If  a  household  had  more  than  one 
interviewed  eligible  youth  (16-  to-20-year-old  PMS  or  PFS  youth)  in  the 
sample,  then  the  parent  base  weight  was  equal  to  the  designated  youth  weight 
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multiplied  by  the  number  of  interviewed  eligible  youths.  Parent  base  weights 
were  computed  for  all  parents  who  responded  to  the  survey.  Nonresponse 
adjustments  via  poststratification  involved  adjustments  by  sex  and  education 
of  parents,  and  by  age,  sex,  and  race/ethnicity  of  youth.  Procedures  for 
parent  weight  calculations  for  the  non- core  and  Perception  modules  were  the 
same  as  those  for  youth. 


Factors  Affecting  the  Precision  of  ACOMS  Sample  Estimates 

A  large  number  of  factors  affect  the  precision  of  sample  estimates.  The 
size  of  the  sample  drawn  is  perhaps  the  most  obvious  factor.  As  sample  sizes 
increase  so  does  the  precision  of  survey  estimates  (though  not  linearly) . 

This  section  discusses  the  impact  of  design  and  unequal  weighting  effects  on 
ACOMS  sample  precision. 


Design  Effects  Due  to  Clustering 

Departures  from  a  SRS  design  such  as  clustering  affect  the  variance  of 
sample  estimates.  Generally  the  effect  is  to  increase  the  variance  of 
estimates  over  that  which  would  be  expected  from  a  SRS  design  of  equal  size. 
Design  effects  are  defined  as 


DEFF 


2  /  2 
O  complex'#  srs 


9 

where  complex  *-s  c^e  variance  of  a  parameter  estimate  from  a  complex  survey 
and  srs  is  the  corresponding  variance  from  a  SRS  design.  Design  effects 
greater  than  1  indicate  the  proportional  increase  in  variance  due  to  complex 
sample  design.  For  example,  a  design  effect  of  1.38  means  that  variance 
estimates  for  a  complex  sample  are  38%  higher  than  those  that  would  be 
obtained  from  a  SRS  of  the  same  size. 

Since  ACOMS  data  were  collected  using  the  RDD  method  the  ACOMS  sample  is  a 
cluster  sample  (with  the  clusters  being  blocks  of  100  telephone  numbers) .  The 
relationship  between  the  variances  of  cluster  (  cluster^  anc*  simPie  random 
samples  (  SRS)  given  by 

^cluster  “  <J2SRS  *  (1  +  P(n  *  1)1 


where  n  is  the  average  number  of  eligible  sample  households  per  cluster  and  p 
(rho)  is  the  intraclass  correlation  among  households  within  clusters.  The 
quantity  {1  +  p(n  -  1)),  then,  is  the  ACOMS  design  effect  due  to  clustering. 

Extensive  evaluation  of  design  effects  due  to  clustering  led  to  the  conclusion 
that  for  the  ACOMS  sample  this  effect  was  minimal.  The  effects  of  clustering 
are  determined  by  two  factors,  n  and  p.  If  the  intraclass  correlation  among 
households  within  clusters  were  .04  (a  value  somewhat  higher  than  generally 
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observed  for  ACOMS)  and  the  average  number  of  eligible  households  20,  then  the 
clustering  effect  would  be  1.76.  A  design  effect  of  this  magnitude  certainly 
cannot  be  ignored.  In  practice,  however,  n  was  usually  less  than  4.  This 
results  in  a  design  effect  of  approximately  1.12.  Since  computed  clustering 
design  effects  were  most  often  even  smaller,  ACOMS  design  effects  due  to 
clustering  were  considered  negligible. 


Unequal  Weighting  Effects 

Although  ACOMS  design  effects  due  to  clustering  proved  to  be  minimal, 
another  design  related  effect  did  demonstrate  a  considerable  impact  on  the 
precision  of  sample  estimates.  This  is  the  inflation  in  variances  due  to 
unequal  survey  sample  weighting  adjustments  (unequal  weighting  effect  or  UWE). 
In  the  ACOMS  design  females  were  selected  at  much  lower  rates  than  males 
generally  and  Hispanic  males  at  rates  much  higher  than  other  males. 
Additionally,  some  questionnaire  modules  were  differentially  administered  to 
subsamples  of  respondents.  Each  of  these  factors  contributed  to  large 
variations  in  sample  adjustment  weights.  Effects  due  to  unequal  weighting 
were  calculated  as 


UWE 


N  *  ZW.2 


where  N  is  the  sample  size  and  the  Wj^  are  the  weights  assigned  to  individual 
respondents.  Calculation  of  the  unequal  weighting  effects  demonstrated  the 
large  impact  that  differential  sample  selection  and  consequently  weighting  had 
upon  the  precision  of  ACOMS  sample  estimates.  The  interpretation  of  UWE  is 
exactly  the  same  as  that  for  DEFF. 

Table  15  shows  the  size  of  the  unequal  weighting  effect  for  various 
sample  groups . 


In  the  majority  of  cases,  design  effects  reflected  expectations 
regarding  the  variability  of  sample  adjustment  weights.  Total  sample  unequal 
weighting  effects,  for  example,  were  higher  than  those  for  males  and  females 
alone.  This  is  due  to  the  homogeneity  of  weights  for  males  and  females 
relative  to  the  total  sample.  For  similar  reasons  males  generally  have  larger 
design  effects  than  females  due  to  Hispanic  male  supplementation. 

The  relationship  between  precision  and  sample  design  and  design  related 
factors  summarized  by  unequal  weighting  effect  highlights  the  related  concept 
of  effective  sample  size.  The  effective  sample  size  of  a  complex  survey  is 
the  SRS  sample  size  required  to  reproduce  the  same  degree  of  precision  in 
sample  estimates.  Effective  sample  size  (ESS)  is  defined  here  as 

obtained  sample  size 
ESS  -  UWE 
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Table  15 

Summary  of  Unequal  Weighting  Effects 


Sample  Group  Unequal  Weighting 

Youth  Sample 


Total  Sample 

2.50 

Males 

1.45 

Females 

1.40 

Questionnaire  Modules  (Youth) 

Core 

2.50 

Perceptions 

Active  Army 

2.60 

Marines 

2.10 

Reserve 

3.30 

Social  Influence 

2.95 

Parent  Sample 

Total  Sample 

2.60 

nale  Parent/Guardian 

1.64 

Female  Parent/Guardian 

1.35 

Effect 
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where  the  obtained  sample  size  is  merely  the  N  of  the  complex  sample.  If,  for 
example,  the  sample  N  is  1,000  and  the  unequal  weighting  effect  is  1.38,  then 
the  effective  sample  size  for  analysis  is  725.  That  is,  the  effective  sample 
size  in  terms  of  the  precision  that  would  be  obtained  from  a  SRS  design  is  725 
or  somewhat  less  than  three-fourths  of  the  obtained  sample  size. 


ACOHS  Statistical  Analyses 


Most  standard  techniques  derived  for  statistical  analysis  (i.e.,  those 
used  by  commonly  used  statistical  software  packages  such  as  SAS,  SPSS,  and 
BMDP)  assume  that  observations  are  independent  and  drawn  using  SRS  selection 
methods.  From  these  assumptions,  classical  statistical  theory  has  developed  a 
wide  variety  of  estimators  valid  under  these  conditions.  For  example,  for  an 
SRS  drawn  without  replacement,  the  sample  mean  of  a  continuous  variable  is 
computed  using  the  familiar  formula: 

_  n 

y  -  l/n£  y± 
i-1 


and  variance  is  calculated  by 


s 


2 


Vn-lZ 


i-1 


Once  a  sample  design  departs  from  SRS,  however,  new  computational 
procedures  are  required  in  order  take  into  account  design  effects  upon 
statistical  estimation.  If,  for  example,  a  population  is  stratified  into  L 
mutually  exclusive  groups  (e.g.,  males  and  females  as  in  ACOMS)  containing  N-^ 
and  N2  individuals  and  random  samples  of  predetermined  sizes  n^  and  ^  are 
drawn  from  each  stratum,  the  standard  (i.e.,  SRS)  formulas  for  mean  and 
variance  become  inappropriate  and  biased.  The  unbias"-?  population  mean  in 
this  case  is  estimated  as: 


L 

y  -  E  wh  yh 
h-1 

where  y^  is  the  mean  of  the  n^  individuals  drawn  from  the  hc^  stratum,  N  is 
the  total  size  of  the  population,  and  -  N^/N.  An  unbiased  estimate  of  y's 
variance  is: 


s2<y>  -  t  «h  a  •  f„>  v»h 

h-1 

2  th 

where  s^  is  the  variance  of  variable  y  in  the  h  stratum,  and  f^  -  n^/N^  is 

the  sampling  fraction  in  the  hc  stratum. 
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As  this  example  illustrates,  the  complexity  of  formulas  and  computations 
required  for  statistical  analysis  increases  as  sample  design  departs  from  SRS 
assumptions.  Two  other  implications  of  complex  designs  for  statistical 
analyses  are  worthy  of  mention  in  connection  with  ACOMS.  First,  regarding 
more  complex  statistics,  in  many  circumstances  it  is  extremely  difficult  to 
even  derive  formulas  for  the  computation  of  estimates  and  variances.  (See 
Cochran,  1963,  for  a  discussion  of  difficulties  encountered  tn  the  regression 
context.)  Second,  certain  sample  selection  techniques  are  such  that  their 
effects  on  the  precision  of  survey  estimates  cannot  be  established  a  priori. 
(See  Goodman  &  Kish,  1950,  for  a  discussion  of  the  technique  of  controlled 
selection. ) 

Consequently,  the  ACOMS  sample  design  which  includes  clustering, 
stratification,  questionnaire  module  rotation,  and  differential  question 
allocation  within  the  Perceptions  module  complicates  analysis.  The  major 
difficulty  encountered  is  in  the  area  of  statistical  significance  testing  and 
variance  estimation.  Unbiased  weighted  statistics  (e.g. ,  means,  proportions, 
correlation  and  regression  coefficients,  etc.)  can  be  produced  by  most 
contemporary  statistical  software  packages.  These  same  packages,  however, 
cannot  produce  unbiased  significance  tests  or  variance  estimates  for  weighted 
survey  data. 

For  ACOMS  analyses,  two  significance  testing  and  variance  estimation 
strategies  were  utilized.  For  contingency  table  analyses  where  chi-square 
tests  were  required,  an  approximate  chi-square  adapted  from  Fellegi  (1980)  was 
used.  In  all  other  circumstances,  the  method  of  balanced  repeated 
replications  (BRR)  was  used  for  variance  estimation  and  significance  testing 
(see  McCarthy,  1966). 


Chi-Square  Approximation  for  Contingency  Table  Analysis 

In  order  to  test  the  statistical  independence  of  row  and  column 
variables  in  a  contingency  table,  an  approximate  chi-square  statistic  was 
computed.  This  approximation  adjusted  weighted  estimates  of  table 
frequencies  and  proportions  to  reflect  the  unweighted  sample  size  for  the 
table  and  survey  sample  unequal  weighting  effects.  The  approximation 
(presented  in  Fellegi,  1980)  was  computed  as: 


where 


t'  ' 


l  £  (Pij  •  Pi-P.j) 

UWF  i,j  Pt  P  j 


(9) 


UWE  - 


t'  ' 


observed  proportion  for  cell  in  row  i 
column  j , 

marginal  proportion  for  row  i, 
marginal  proportion  for  column  j , 
expected  proportion  for  cell  in  row  i 
column  j , 

the  average  unequal  weighting  effect  for  the 
variables  i  and  j ,  and 

is  distributed  approximately  as  chi-square. 
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The  effect  of  this  approximation  is  to  compute  chi-square  using  weighted 
observed  and  expected  proportions  while  simultaneously  adjusting  the  computed 
chi-square  statistic  to  reflect  the  effective  (i.e.,  not  weighted  or  SRS) 
sample  size.  The  computation  of  unequal  weighting  effect  was  accomplished 
using  the  technique  discussed  in  the  previous  section  of  this  chapter. 
Performance  of  the  chi-square  significance  test  was  carried  out  within  SAS  and 
personal  computer  (PC)  spreadsheet  environments  as  convenience  dictated. 


Balanced  Repeated  Replication  CBRR)  Variance  Estimation 

ACOMS  analyses  required  the  production  of  numerous  statistics  (e.g., 
means,  correlations,  and  regression  coefficients,  etc.).  For  contingency 
table  analyses  an  approximate  chi-square  test  was  used  to  evaluate  statistical 
independence.  For  the  remaining  tests  of  statistical  significance,  the  BRR 
methodology  was  used. 

Essentially,  BRR  empirically  mimics  the  sampling  distribution  theory 
which  underlies  variance  estimation  formulas  derived  for  SRS  samples  (i.e., 
the  sampling  variance  that  would  be  expected  to  occur  were  an  infinite  number 
of  samples  drawn  from  the  population).  The  BRR  methodology  used  the  full 
sample  statistic  estimate  as  a  benchmark  and  repeatedly  drew  half-samples 
(i.e.,  samples  one-half  the  size  of  the  full  sample  -  for  ACOMS,  68  half¬ 
samples  were  drawn)  from  this  sample.  Sample  variance  of  the  statistic  (e.g., 
a  mean,  proportion,  correlation  coefficient,  regression  coefficient,  etc.) 
were  then  computed  as  the  variance  of  half -sample  estimates  about  full  sample 
estimates.  Computationally,  ACOMS  variance  estimates  were  obtained  using  the 
following  formula: 


68 

s2(e)  -  1/68  £  (Er  -  Ef ) 2  (10) 

r-1 


where 


s2(e) 


estimated  sample  variance  of  statistic  E, 
the  estimated  value  of  the  statistic  for 
the  full  sample,  and 
the  estimate  for  the  rc^  half-sample 
replicate . 


As  discussed,  this  structuring  of  variance  estimation  and  significance 
testing  has  the  advantage  of  mimicking  classical  sampling  distribution  theory 
and  is,  therefore,  conceptually  straightforward.  Additionally,  by  using  half¬ 
samples  drawn  from  the  complex  sample,  the  sample  design  and  adjustment 
weights  are  explicitly  incorporated  in  the  BRR  methodology.  Therefore, 
variance  estimation  and  significance  testing  did  not  require  the  separate 
estimation  of  quantities  such  as  design  or  unequal  weighting  effects. 

"Balanced"  within  the  context  of  BRR  requires  that  the  half-samples 
drawn  for  the  purpose  of  variance  estimation  are  uncorrelated.  The  ACOMS 
database  has  been  structured  to  include  variables  identifying  all  replicate 
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samples  formed  according  to  this  statistical  requirement.  In  addition,  the 
weights  necessary  for  BRR  half-sample  estimation  were  included  on  the 
database.  Directly  analogous  weights  and  samples  have  been  established  for 
the  parental  sample  (the  major  distinction  being  that  32  rather  than  68  half- 
samples  are  used  for  estimation  purposes). 
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